Original: D IFFERENTIABLE O PTIMIZATION OF G ENERALIZED N ONDECOMPOSABLE F UNCTIONS USING L INEAR P ROGRAMS
Revised: D IFFERENTIABLE O PTIMIZATION OF G ENERALIZED N ONDECOMPOSABLE F UNCTIONS USING L INEAR P ROGRAMS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose a framework which makes it feasible to directly train deep neural networks with respect to popular families of task-speciﬁc non-decomposable performance measures such as AUC, multi-class AUC, F -measure and others.
Revised: We propose a framework which makes it feasible to directly train deep neural networks with respect to popular families of task-speciﬁc non-decomposable performance measures such as AUC, multi-class AUC, F -measure and others.

Original: A common feature of the optimization model that emerges from these tasks is that it involves solving a Linear Programs (LP) during training where representations learned by upstream layers inﬂuence the constraints.
Revised: A common feature of the optimization model that emerges from these tasks is that it involves solving a Linear Programs (LP) during training where representations learned by upstream layers inﬂuence the constraints.

Original: The constraint matrix is not only large but the constraints are also modiﬁed at each iteration.
Revised: The constraint matrix is not only large but the constraints are also modiﬁed at each iteration.

Original: We show how adopting a set of inﬂuential ideas proposed by Mangasarian for 1-norm SVMs – which advocates for solving LPs with a generalized Newton method – provides a simple and effective solution.
Revised: We show how adopting a set of inﬂuential ideas proposed by Mangasarian for 1-norm SVMs – which advocates for solving LPs with a generalized Newton method – provides a simple and effective solution.

Original: In particular, this strategy needs little unrolling, which makes it more efﬁcient during backward pass.
Revised: In particular, this strategy needs little unrolling, which makes it more efﬁcient during backward pass.

Original: While a number of specialized algorithms have been proposed for the models that we describe here, our module turns out to be applicable without any speciﬁc adjustments or relaxations.
Revised: While a number of specialized algorithms have been proposed for the models that we describe here, our module turns out to be applicable without any speciﬁc adjustments or relaxations.

Original: We describe each use case, study its properties and demonstrate the efﬁcacy of the approach over alternatives which use surrogate lower bounds and often, specialized optimization schemes.
Revised: We describe each use case, study its properties and demonstrate the efﬁcacy of the approach over alternatives which use surrogate lower bounds and often, specialized optimization schemes.

Original: Frequently, we achieve superior computational behavior and performance improvements on common datasets used in the literature
Revised: Frequently, we achieve superior computational behavior and performance improvements on common datasets used in the literature

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Commonly used losses such as cross-entropy used in deep neural network (DNN) models can be expressed as a sum over the per-sample losses incurred by the current estimate of the model.
Revised: Commonly used losses such as cross-entropy used in deep neural network (DNN) models can be expressed as a sum over the per-sample losses incurred by the current estimate of the model.

Original: This allows the direct use of mature optimization routines, and is sufﬁcient for a majority of use cases.
Revised: This allows the direct use of mature optimization routines, and is sufﬁcient for a majority of use cases.

Original: But in various applications ranging from ranking/retrieval systems to class imbalanced learning, the most suitable losses for the task do not admit a “decompose over samples” form.
Revised: But in various applications ranging from ranking/retrieval systems to class imbalanced learning, the most suitable losses for the task do not admit a “decompose over samples” form.

Original: Examples include
Revised: Examples include

Original: Area under the ROC curve (AUC), multi-class variants of AUC, F -score, Precision at a ﬁxed recall (P@R) and others.
Revised: Area under the ROC curve (AUC), multi-class variants of AUC, F -score, Precision at a ﬁxed recall (P@R) and others.

Original: Optimizing such measures in a scalable manner can pose challenges even in the shallow setting.
Revised: Optimizing such measures in a scalable manner can pose challenges even in the shallow setting.

Original: Since the compromise involved in falling back on a decomposable loss when a non-decomposable objective may be more appropriate for the task at hand can range from negligible to concerning depending on the application, the last few years have seen a number of interesting approaches proposed to efﬁciently deal with such structured and non-decomposable losses.
Revised: Since the compromise involved in falling back on a decomposable loss when a non-decomposable objective may be more appropriate for the task at hand can range from negligible to concerning depending on the application, the last few years have seen a number of interesting approaches proposed to efﬁciently deal with such structured and non-decomposable losses.

Original: To this end, recent algorithms for AUC maximization have been developed based on convex surrogate losses
Revised: To this end, recent algorithms for AUC maximization have been developed based on convex surrogate losses

Original: Liu et al.
Revised: Liu et al.

Original: ; Natole et al.
Revised: ; Natole et al.

Original: (2018) in a linear model or in conjuction with a deep neural network Liu et al.
Revised: (2018) in a linear model or in conjuction with a deep neural network Liu et al.

Original: (2019) as well as stochastic and online variations (Ataman et al.
Revised: (2019) as well as stochastic and online variations (Ataman et al.

Original: (2006); Cortes & Mohri (2004);
Revised: (2006); Cortes & Mohri (2004);

Original: Gao et al.
Revised: Gao et al.

Original: Liu et al.
Revised: Liu et al.

Original: (2018; 2019)) are available.
Revised: (2018; 2019)) are available.

Original: Methods for measures other than the AUC have also been studied – exact algorithms for optimizing F -score Nan et al.
Revised: Methods for measures other than the AUC have also been studied – exact algorithms for optimizing F -score Nan et al.

Original: ; Dembczynski et al.
Revised: ; Dembczynski et al.

Original: (2011), optimizing average precision through direct optimization Song et al.
Revised: (2011), optimizing average precision through direct optimization Song et al.

Original: scalable methods for non-decomposable objectives Eban et al.
Revised: scalable methods for non-decomposable objectives Eban et al.

Original: Venkatesh et al.
Revised: Venkatesh et al.

Original: and using a structured hinge-loss upper bound to optimize average precision and NDCG Mohapatra et al.
Revised: and using a structured hinge-loss upper bound to optimize average precision and NDCG Mohapatra et al.

Original: Recently, the AP-Perf method Fathony & Kolter (2020) showed how custom non-decomposable performance measures can be conveniently incorporated into differentiable pipelines.
Revised: Recently, the AP-Perf method Fathony & Kolter (2020) showed how custom non-decomposable performance measures can be conveniently incorporated into differentiable pipelines.

Original: It is known that a number of these non-decomposable objectives can be expressed in the form of an integer program that can be relaxed to a linear program (LP).
Revised: It is known that a number of these non-decomposable objectives can be expressed in the form of an integer program that can be relaxed to a linear program (LP).

Original: Earlier approaches adapted methods for structured SVMs Joachims et al.
Revised: Earlier approaches adapted methods for structured SVMs Joachims et al.

Original: or cutting plane techniques Yue et al.
Revised: or cutting plane techniques Yue et al.

Original: and were interesting but had difﬁculty scaling to larger datasets.
Revised: and were interesting but had difﬁculty scaling to larger datasets.

Original: More recently, strategies have instead focused on the stochastic setting where we operate on a mini-batch of samples.
Revised: More recently, strategies have instead focused on the stochastic setting where we operate on a mini-batch of samples.

Original: A common strategy, which is generally efﬁcient, is to study the combinatorial form of one or more losses of interest and derive surrogate lower bounds.
Revised: A common strategy, which is generally efﬁcient, is to study the combinatorial form of one or more losses of interest and derive surrogate lower bounds.

Original: These are then tackled via specialized optimization routines.
Revised: These are then tackled via specialized optimization routines.

Original: The arguably more direct alternative of optimizating as a module embedded in a DNN architecture remained difﬁcult until recently – but OptNet Amos & Kolter (2017) and CVXPY now offer support for solving certain objectives as differentiable layers within a network.
Revised: The arguably more direct alternative of optimizating as a module embedded in a DNN architecture remained difﬁcult until recently – but OptNet Amos & Kolter (2017) and CVXPY now offer support for solving certain objectives as differentiable layers within a network.

Original: Further, methods based on implicit differentiation have been applied to various problems, showing impressive performance.
Revised: Further, methods based on implicit differentiation have been applied to various problems, showing impressive performance.

Original: Our approach is based on the premise that tackling the LP form of the non-decomposable objective as a module within the DNN, one which permits forward and reverse mode differentiation and can utilize in-built support for specialized GPU hardware in modern libraries such as PyTorch, is desirable.
Revised: Our approach is based on the premise that tackling the LP form of the non-decomposable objective as a module within the DNN, one which permits forward and reverse mode differentiation and can utilize in-built support for specialized GPU hardware in modern libraries such as PyTorch, is desirable.

Original: First, as long as a suitable LP formulation for an objective is available, the module may be directly used.
Revised: First, as long as a suitable LP formulation for an objective is available, the module may be directly used.

Original: Second, based on which scheme is used to solve the LP, one may be able to provide guarantees for the non-decomposable objective based on simple calculations (e.g., number of constraints, primal-dual gap).
Revised: Second, based on which scheme is used to solve the LP, one may be able to provide guarantees for the non-decomposable objective based on simple calculations (e.g., number of constraints, primal-dual gap).

Original: The current tools, however, do not entirely address all these requirements, as we brieﬂy describe next.
Revised: The current tools, however, do not entirely address all these requirements, as we brieﬂy describe next.

Original: A speciﬁc characteristic of the LPs that arise from the losses mentioned above is that the constraints are modiﬁed at each iteration – as a function of the updates to the representations of the data in the upstream layers.
Revised: A speciﬁc characteristic of the LPs that arise from the losses mentioned above is that the constraints are modiﬁed at each iteration – as a function of the updates to the representations of the data in the upstream layers.

Original: Further, the mini-batch of samples changes at each iteration.
Revised: Further, the mini-batch of samples changes at each iteration.

Original: Solvers within CVXPY, are effective but due to their general-purpose nature, rely on interior point methods.
Revised: Solvers within CVXPY, are effective but due to their general-purpose nature, rely on interior point methods.

Original: OptNet is quite efﬁcient but designed for quadratic programs (QP): the theoretical results and its efﬁciency depends on factorizing a matrix in the quadratic term in the objective (which is zero/noninvertible for LPs).
Revised: OptNet is quite efﬁcient but designed for quadratic programs (QP): the theoretical results and its efﬁciency depends on factorizing a matrix in the quadratic term in the objective (which is zero/noninvertible for LPs).

Original: The primal-dual properties and implicit differentiation for QPs do not easily translate to LPs, and efﬁcient ways of dealing with constraints that are iteratively updated are not widely available at this time.
Revised: The primal-dual properties and implicit differentiation for QPs do not easily translate to LPs, and efﬁcient ways of dealing with constraints that are iteratively updated are not widely available at this time.

Original: In principle, of course, backpropagating through a convex optimization model (and in particular, LPs) is not an unsolved problem.
Revised: In principle, of course, backpropagating through a convex optimization model (and in particular, LPs) is not an unsolved problem.

Original: For LPs, we can take derivatives of the optimal value (or the optimal solution) of the model with respect to the LP parameters, and this can be accomplished by calling a powerful external solver.
Revised: For LPs, we can take derivatives of the optimal value (or the optimal solution) of the model with respect to the LP parameters, and this can be accomplished by calling a powerful external solver.

Original: Often, this would involve running the solver on the CPU, which introduces overhead. .
Revised: Often, this would involve running the solver on the CPU, which introduces overhead. .

Original: The ideas in Meng et al.
Revised: The ideas in Meng et al.

Original: (2020) are relevant in that the optimization steps for the LP are unrolled and only involve simple linear algebra operations but the formulation is applicable when the number of constraints are about the same as the number of variables – an assumption that does not hold for the models we will study.
Revised: (2020) are relevant in that the optimization steps for the LP are unrolled and only involve simple linear algebra operations but the formulation is applicable when the number of constraints are about the same as the number of variables – an assumption that does not hold for the models we will study.

Original: In §3, we show that the modiﬁed Newton’s algorithm in Mangasarian (2004) can be used for deep neural network (DNN) training in an end-to-end manner without requiring an external solvers where support for GPUs remains limited.
Revised: In §3, we show that the modiﬁed Newton’s algorithm in Mangasarian (2004) can be used for deep neural network (DNN) training in an end-to-end manner without requiring an external solvers where support for GPUs remains limited.

Original: Speciﬁcally, by exploiting self-concordance of the objective, we show that the algorithm can converge globally without line search strategies.
Revised: Speciﬁcally, by exploiting self-concordance of the objective, we show that the algorithm can converge globally without line search strategies.

Original: On the practical side, we analyze the gradient properties, and some modiﬁcations to improve stability during backpropagation.
Revised: On the practical side, we analyze the gradient properties, and some modiﬁcations to improve stability during backpropagation.

Original: We show that this scheme based on Mangasarian’s parametric exterior penalty formulation of the primal LP can be a computationally effective and scalable strategy to solve LPs with a large number of constraints.
Revised: We show that this scheme based on Mangasarian’s parametric exterior penalty formulation of the primal LP can be a computationally effective and scalable strategy to solve LPs with a large number of constraints.

Original:  N ONDECOMPOSABLE F UNCTIONS AND CORRESPONDING LP MODELS
Revised:  N ONDECOMPOSABLE F UNCTIONS AND CORRESPONDING LP MODELS

Original: We ﬁrst present a standard LP form and then reparameterize several generalized nondecomposable objectives in this way, summarized in Table 5 in the appendix.
Revised: We ﬁrst present a standard LP form and then reparameterize several generalized nondecomposable objectives in this way, summarized in Table 5 in the appendix.

Original: We start with the binary AUC, extend it to multi-class AUC, and then later, show a ratio objective, F -score.
Revised: We start with the binary AUC, extend it to multi-class AUC, and then later, show a ratio objective, F -score.

Original: Some other objectives are described in the appendix.
Revised: Some other objectives are described in the appendix.

Original: 2. N OTATIONS AND G ENERALIZED LP FORMULATION
Revised: 2. N OTATIONS AND G ENERALIZED LP FORMULATION

Original: Notations.
Revised: Notations.

Original: We use the following notations:
Revised: We use the following notations:

Original: (i) n : number of samples used in training.
Revised: (i) n : number of samples used in training.

Original: (ii) X ∈ R n × d : the explanatory features fed to a classiﬁer (e.g., parameterized by w ); (iii) f
Revised: (ii) X ∈ R n × d : the explanatory features fed to a classiﬁer (e.g., parameterized by w ); (iii) f

Original: ( x i ) (or f ( i ) ): a score function for the classiﬁer where x i ∈
Revised: ( x i ) (or f ( i ) ): a score function for the classiﬁer where x i ∈

Original: X such that f ( X ) =
Revised: X such that f ( X ) =

Original: w X ; (iv) Y ∈ { 0 , 1 } : target label and ˆ Y ∈ { 0 , 1 } : predicted label for binary classiﬁcation, both in R n ; (v) φ ( · ) : non linear function applied on f ( X ) ; (vi) A ⊗ B : Kronecker product of matrices A and B .
Revised: w X ; (iv) Y ∈ { 0 , 1 } : target label and ˆ Y ∈ { 0 , 1 } : predicted label for binary classiﬁcation, both in R n ; (v) φ ( · ) : non linear function applied on f ( X ) ; (vi) A ⊗ B : Kronecker product of matrices A and B .

Original: (vii) I r : Identity matrix of size r and 1 is the indicator function.
Revised: (vii) I r : Identity matrix of size r and 1 is the indicator function.

Original: (viii) B k, (cid:112) (and B | ,k (cid:48) ) gives the k -th row (and k (cid:48) -th) column of B .
Revised: (viii) B k, (cid:112) (and B | ,k (cid:48) ) gives the k -th row (and k (cid:48) -th) column of B .

Original: LP formulation.
Revised: LP formulation.

Original: We consider a general linear program (LP) that contains nonnegative variables as well as inequality and equality constraints.
Revised: We consider a general linear program (LP) that contains nonnegative variables as well as inequality and equality constraints.

Original: The form of the LP is given as
Revised: The form of the LP is given as

Original: [Equation]
Revised: [Equation]

Original: We can write it more succinctly as
Revised: We can write it more succinctly as

Original: [Equation]
Revised: [Equation]

Original: The corresponding primal LP becomes min x c T x s.t Ax ≤ b, x ≥ 0 .
Revised: The corresponding primal LP becomes min x c T x s.t Ax ≤ b, x ≥ 0 .

Original: 2. M AXIMIZING AUC
Revised: 2. M AXIMIZING AUC

Original: The Area under the ROC Curve (AUC) calculates the probability that a classiﬁer f ( · ) will rank a randomly chosen positive sample higher than a randomly chosen negative sample.
Revised: The Area under the ROC Curve (AUC) calculates the probability that a classiﬁer f ( · ) will rank a randomly chosen positive sample higher than a randomly chosen negative sample.

Original: Since AUC varies between 0 and 1 , where 1 represents all positives being ranked above the negatives.
Revised: Since AUC varies between 0 and 1 , where 1 represents all positives being ranked above the negatives.

Original: AUC may be estimated using the Wilcoxon-Mann-Whitney (WMW)
Revised: AUC may be estimated using the Wilcoxon-Mann-Whitney (WMW)

Original: Statistic Hanley & McNeil (1982), as
Revised: Statistic Hanley & McNeil (1982), as

Original: Deﬁnition 2.1 (AUC) .
Revised: Deﬁnition 2.1 (AUC) .

Original: Let n be the number of samples.
Revised: Let n be the number of samples.

Original: Let X
Revised: Let X

Original: + (and X − resp.)
Revised: + (and X − resp.)

Original: be the set of positive (and negative resp.)
Revised: be the set of positive (and negative resp.)

Original: samples such that | X + | + | X − | = n where | · | is the cardinality of the set.
Revised: samples such that | X + | + | X − | = n where | · | is the cardinality of the set.

Original: Then,
Revised: Then,

Original: AUC is given as (cid:0) (cid:80)
Revised: AUC is given as (cid:0) (cid:80)

Original: [Equation]
Revised: [Equation]

Original: X + | | X − | (cid:1) for x i : i ∈ { 1 , · · · , n } .
Revised: X + | | X − | (cid:1) for x i : i ∈ { 1 , · · · , n } .

Original: Here, we follow Ataman et al.
Revised: Here, we follow Ataman et al.

Original: (2006) to calculate the AUC based on the WMW statistic as follows.
Revised: (2006) to calculate the AUC based on the WMW statistic as follows.

Original: [Equation]
Revised: [Equation]

Original: x i ∈ X
Revised: x i ∈ X

Original: + , x j ∈ X −
Revised: + , x j ∈ X −

Original: [Equation]
Revised: [Equation]

Original: ij ≥
Revised: ij ≥

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where (cid:15) is a given constant.
Revised: where (cid:15) is a given constant.

Original: The model in (2), maximizes AUC indirectly, by minimizing the number of pairs (one each from the positive and negative classes) where the positive sample is not ranked higher than the negative sample: so, the number of zero entries in z equals the number of pairs where this condition is not true.
Revised: The model in (2), maximizes AUC indirectly, by minimizing the number of pairs (one each from the positive and negative classes) where the positive sample is not ranked higher than the negative sample: so, the number of zero entries in z equals the number of pairs where this condition is not true.

Original: To compute the AUC, we do:
Revised: To compute the AUC, we do:

Original: [Equation]
Revised: [Equation]

Original: If z ij is 0 , then (cid:15) − 1 relu (0 , − z ij + (cid:15) ) equals 1 .
Revised: If z ij is 0 , then (cid:15) − 1 relu (0 , − z ij + (cid:15) ) equals 1 .

Original: Otherwise z ij > 0 , it follows from the ﬁrst constraint in (2), that z ij ≥ (cid:15) , so (cid:15) − 1 relu (0 , − z ij + (cid:15) ) equals 0 .
Revised: Otherwise z ij > 0 , it follows from the ﬁrst constraint in (2), that z ij ≥ (cid:15) , so (cid:15) − 1 relu (0 , − z ij + (cid:15) ) equals 0 .

Original: Observe that in (2), the number of constraints is | X || X − | , which is quadratic in n .
Revised: Observe that in (2), the number of constraints is | X || X − | , which is quadratic in n .

Original: 2. M AXIMIZING M ULTI - CLASS AUC
Revised: 2. M AXIMIZING M ULTI - CLASS AUC

Original: An extension of AUC to the multi-class setting, AUC µ , is deﬁned in Kleiman & Page (2019).
Revised: An extension of AUC to the multi-class setting, AUC µ , is deﬁned in Kleiman & Page (2019).

Original: The
Revised: The

Original: AUC µ objective optimizes an indicator matrix computed on the orientation function, O i,j deﬁned as,
Revised: AUC µ objective optimizes an indicator matrix computed on the orientation function, O i,j deﬁned as,

Original: Deﬁnition 2.2 (Orientation Function; Kleiman & Page (2019)) .
Revised: Deﬁnition 2.2 (Orientation Function; Kleiman & Page (2019)) .

Original: Assume we have K classes { y 1 , · · · , y K } .
Revised: Assume we have K classes { y 1 , · · · , y K } .

Original: Let f ( x i , (cid:112) )
Revised: Let f ( x i , (cid:112) )

Original: ∈ R K indicate the model’s prediction on x i for each of the K classes (class-speciﬁc probability) 1 .
Revised: ∈ R K indicate the model’s prediction on x i for each of the K classes (class-speciﬁc probability) 1 .

Original: Let x ∗ i provide the index of x i ’s true class label.
Revised: Let x ∗ i provide the index of x i ’s true class label.

Original: Let P ∈ R k × k be a partition matrix where P k,k (cid:48) is the cost of classifying a sample as class k when it should be k (cid:48) .
Revised: Let P ∈ R k × k be a partition matrix where P k,k (cid:48) is the cost of classifying a sample as class k when it should be k (cid:48) .

Original: Deﬁne v kk (cid:48)
Revised: Deﬁne v kk (cid:48)

Original: =
Revised: =

Original: P k, (cid:112) − P k (cid:48) , (cid:112) and (cid:101) v = v x ∗ i x ∗ j ∈ R K .
Revised: P k, (cid:112) − P k (cid:48) , (cid:112) and (cid:101) v = v x ∗ i x ∗ j ∈ R K .

Original: Then, O i,j = ( (cid:101) v x ∗ i − (cid:101) v x ∗ j )( (cid:104) (cid:101) v , f ( x i , (cid:112) )
Revised: Then, O i,j = ( (cid:101) v x ∗ i − (cid:101) v x ∗ j )( (cid:104) (cid:101) v , f ( x i , (cid:112) )

Original: (cid:105)−(cid:104) (cid:101) v , f ( x j , (cid:112) )
Revised: (cid:105)−(cid:104) (cid:101) v , f ( x j , (cid:112) )

Original: (cid:105) ).
Revised: (cid:105) ).

Original: Here, v kk (cid:48) ranks the instances by their cost difference between assignments to class k and k (cid:48) .
Revised: Here, v kk (cid:48) ranks the instances by their cost difference between assignments to class k and k (cid:48) .

Original: For two classes, say indexed by 1 and 2 , v 1 , 2 · f = 0 is the decision boundary between the classes.
Revised: For two classes, say indexed by 1 and 2 , v 1 , 2 · f = 0 is the decision boundary between the classes.

Original: The ranking is correct for x i and x j (with correct labels 1 and 2 resp.)
Revised: The ranking is correct for x i and x j (with correct labels 1 and 2 resp.)

Original: if the orientation of the points w.r.t.
Revised: if the orientation of the points w.r.t.

Original: the decision boundary is the same as those of the class labels (converted to one-hot vectors).
Revised: the decision boundary is the same as those of the class labels (converted to one-hot vectors).

Original: Then AUC µ = (cid:80) k<k (cid:48) g k,k (cid:48) (cid:80) i ∈
Revised: Then AUC µ = (cid:80) k<k (cid:48) g k,k (cid:48) (cid:80) i ∈

Original: D k ,j ∈ D (cid:48)
Revised: D k ,j ∈ D (cid:48)

Original: k
Revised: k

Original: 1 O i,j ≥ 0 , where D k is the index set of samples having same class label k and g is a constant derived from class sizes.
Revised: 1 O i,j ≥ 0 , where D k is the index set of samples having same class label k and g is a constant derived from class sizes.

Original: In formulating the linear program, we observe that the model is dictated on how P is deﬁned.
Revised: In formulating the linear program, we observe that the model is dictated on how P is deﬁned.

Original: One way is to set P ( k, k ) =
Revised: One way is to set P ( k, k ) =

Original: 0 ∀ k and 1 for all other entries.
Revised: 0 ∀ k and 1 for all other entries.

Original: We can also deﬁne a P with arbitary entries or formulate AUC in a one-vs-all setting.
Revised: We can also deﬁne a P with arbitary entries or formulate AUC in a one-vs-all setting.

Original: Here, we show the model in the ﬁrst case (the other two cases are discussed
Revised: Here, we show the model in the ﬁrst case (the other two cases are discussed

Original: Appendix A.1).
Revised: Appendix A.1).

Original: Similar to our previous formulation, the goal is to minimize the sum of negative values of the orientation function, rather than maximize the sum of positive values.
Revised: Similar to our previous formulation, the goal is to minimize the sum of negative values of the orientation function, rather than maximize the sum of positive values.

Original: Let (cid:101) f
Revised: Let (cid:101) f

Original: ( i, j ) =
Revised: ( i, j ) =

Original: f ( x i , x ∗ i ) − f ( x j , x ∗ i ) + f ( x j , x ∗ j ) − f ( x i , x ∗ i ) .
Revised: f ( x i , x ∗ i ) − f ( x j , x ∗ i ) + f ( x j , x ∗ j ) − f ( x i , x ∗ i ) .

Original: Then the LP formulation is.
Revised: Then the LP formulation is.

Original: [Equation]
Revised: [Equation]

Original: z ij
Revised: z ij

Original: [Equation]
Revised: [Equation]

Original: Here (cid:101) d ij = (cid:101) v x ∗
Revised: Here (cid:101) d ij = (cid:101) v x ∗

Original: i − (cid:101) v x ∗ j .
Revised: i − (cid:101) v x ∗ j .

Original: This can be seen as an extension of our binary AUC model, where the z ij is the ranking between a pair of points deﬁned for multiple classes.
Revised: This can be seen as an extension of our binary AUC model, where the z ij is the ranking between a pair of points deﬁned for multiple classes.

Original: 2. M AXIMIZING F - SCORE
Revised: 2. M AXIMIZING F - SCORE

Original: The F -score (or F -measure) is a representative of objectives expressed as ratios of some combination of True positives (TP), False positives (FP), True negatives (TN) and False negatives (FN).
Revised: The F -score (or F -measure) is a representative of objectives expressed as ratios of some combination of True positives (TP), False positives (FP), True negatives (TN) and False negatives (FN).

Original: The general form of the ratio functions and formulations for other objectives is in the Appendix A.2.
Revised: The general form of the ratio functions and formulations for other objectives is in the Appendix A.2.

Original: Speciﬁcally, F -score is deﬁned as follows:
Revised: Speciﬁcally, F -score is deﬁned as follows:

Original: Deﬁnition 2.3 ( F -
Revised: Deﬁnition 2.3 ( F -

Original: [Equation]
Revised: [Equation]

Original: F
Revised: F

Original: [Equation]
Revised: [Equation]

Original: The second part of the equality comes from simplying the precision (
Revised: The second part of the equality comes from simplying the precision (

Original: TP TP + FP ) and recall (
Revised: TP TP + FP ) and recall (

Original: [Equation]
Revised: [Equation]

Original: TP + FN ) based formula Dembczynski et al.
Revised: TP + FN ) based formula Dembczynski et al.

Original: The last part is obtained by replacing TP with ( Y T × ˆ Y ),
Revised: The last part is obtained by replacing TP with ( Y T × ˆ Y ),

Original: FP with (1 − Y ) T × ˆ Y and FN with ( Y ) T × (1 − ˆ Y ) as functions of Y and ˆ Y .
Revised: FP with (1 − Y ) T × ˆ Y and FN with ( Y ) T × (1 − ˆ Y ) as functions of Y and ˆ Y .

Original: This leads to the following integer fractional optimization model,
Revised: This leads to the following integer fractional optimization model,

Original: [Equation]
Revised: [Equation]

Original: To solve this, we ﬁrst relax the constraint on ˆ Y and reformulate the model as the following LP, by introducing two variables z ∈ R n and t ∈ R 1 where z =
Revised: To solve this, we ﬁrst relax the constraint on ˆ Y and reformulate the model as the following LP, by introducing two variables z ∈ R n and t ∈ R 1 where z =

Original: [Equation]
Revised: [Equation]

Original: Y + b , t =
Revised: Y + b , t =

Original: [Equation]
Revised: [Equation]

Original: 1 T ˆ Y + b
Revised: 1 T ˆ Y + b

Original: and i ∈ { 1 , · · · , n } :
Revised: and i ∈ { 1 , · · · , n } :

Original: [Equation]
Revised: [Equation]

Original: Remark 1.
Revised: Remark 1.

Original: (a) ensures the appropriate relation between z , t and ˆ Y and is essentially a reformulaton of the ratio objective as a linear function with a ﬁxed denominator.
Revised: (a) ensures the appropriate relation between z , t and ˆ Y and is essentially a reformulaton of the ratio objective as a linear function with a ﬁxed denominator.

Original: ˆ Y is recovered from the solution to the linear program by computing ˆ Y i =
Revised: ˆ Y is recovered from the solution to the linear program by computing ˆ Y i =

Original: z i t when t > 0 and ˆ Y i
Revised: z i t when t > 0 and ˆ Y i

Original: =
Revised: =

Original: z i otherwise.
Revised: z i otherwise.

Original: (b) sets an upper bound for ˆ Y i ≤ 1 .
Revised: (b) sets an upper bound for ˆ Y i ≤ 1 .

Original: (c) ties the output of the previous layer φ ( f ( x i ))
Revised: (c) ties the output of the previous layer φ ( f ( x i ))

Original: (a classiﬁer score for ˆ Y i , see the deﬁnition in Section 2.1) as a input in this layer.
Revised: (a classiﬁer score for ˆ Y i , see the deﬁnition in Section 2.1) as a input in this layer.

Original: Assume φ ( . ) ∈ {− 1 , 1 } (ensured if φ is sigmoid or tanh) is the indicator of the class label (based on sign).
Revised: Assume φ ( . ) ∈ {− 1 , 1 } (ensured if φ is sigmoid or tanh) is the indicator of the class label (based on sign).

Original: We want
Revised: We want

Original: ˆ Y
Revised: ˆ Y

Original: i ≥ φ ( f ( x i )) and ˆ Y i ≤ 1 + φ ( f ( x i )) .
Revised: i ≥ φ ( f ( x i )) and ˆ Y i ≤ 1 + φ ( f ( x i )) .

Original: If ˆ Y i is { 0 , 1 } , these two constraints ensures that ˆ Y i = 0 when φ ( f ( x i ))
Revised: If ˆ Y i is { 0 , 1 } , these two constraints ensures that ˆ Y i = 0 when φ ( f ( x i ))

Original: ≤and ˆ Y i = 

Revised: ≤and ˆ Y i =

Original: when φ ( f ( x i )) > 0 .
Revised: when φ ( f ( x i )) > 0 .

Original: When ˆ Y i is relaxed and replaced with z and t , we get the equivalent form (c).
Revised: When ˆ Y i is relaxed and replaced with z and t , we get the equivalent form (c).

Original: This model imposes 4 n constraints for n samples.
Revised: This model imposes 4 n constraints for n samples.

Original: Since this is a maximization, a solution to the LP,
Revised: Since this is a maximization, a solution to the LP,

Original: O ˆ Y , is an upper bound on the integer objective opt ∗ , and serves as the loss.
Revised: O ˆ Y , is an upper bound on the integer objective opt ∗ , and serves as the loss.

Original:  B ACKPROPAGATION VIA F AST E XTERIOR P ENALTY O PTIMIZATION
Revised:  B ACKPROPAGATION VIA F AST E XTERIOR P ENALTY O PTIMIZATION

Original: Unlike traditional feedforward networks, where the output of each layer is a relatively simple (though non-linear) function of the previous layer, a LP layer must solve a constrained optimization problem, therefore implementing scalable and efﬁcient backpropagation schemes that minimizes computational overhead requires more care and as reviewed in §1, is an active topic of research.
Revised: Unlike traditional feedforward networks, where the output of each layer is a relatively simple (though non-linear) function of the previous layer, a LP layer must solve a constrained optimization problem, therefore implementing scalable and efﬁcient backpropagation schemes that minimizes computational overhead requires more care and as reviewed in §1, is an active topic of research.

Original: This problem is, of course, not unique to LPs and manifests in differentiable sorting Mena et al.
Revised: This problem is, of course, not unique to LPs and manifests in differentiable sorting Mena et al.

Original: and formulating quadratic or cone programs Amos et al.
Revised: and formulating quadratic or cone programs Amos et al.

Original: One may unroll gradient descent steps Amos et al.
Revised: One may unroll gradient descent steps Amos et al.

Original: (2017); Goodfellow et al.
Revised: (2017); Goodfellow et al.

Original: ; Metz et al.
Revised: ; Metz et al.

Original: or use projections Zeng et al.
Revised: or use projections Zeng et al.

Original: Recently
Revised: Recently

Original: Agrawal et al.
Revised: Agrawal et al.

Original: (2019) introduced a package for differentiable constrained convex programming, which includes LPs as a special case.
Revised: (2019) introduced a package for differentiable constrained convex programming, which includes LPs as a special case.

Original: For LPs, Meng et al.
Revised: For LPs, Meng et al.

Original: (2020) presents an unrolled scheme and Blondel et al.
Revised: (2020) presents an unrolled scheme and Blondel et al.

Original: (2020) shows that we can differentiate through LP formulations of sorting/ranking exactly by using smooth approximations of projection steps.
Revised: (2020) shows that we can differentiate through LP formulations of sorting/ranking exactly by using smooth approximations of projection steps.

Original: Berthet et al.
Revised: Berthet et al.

Original: (2020) describes an interesting approach where one computes approximate gradients through ranking/shortest path problems by stochastic perturbation techniques.
Revised: (2020) describes an interesting approach where one computes approximate gradients through ranking/shortest path problems by stochastic perturbation techniques.

Original: Remark 2.
Revised: Remark 2.

Original: Some previous works Zeng et al.
Revised: Some previous works Zeng et al.

Original: (2019) have considered LPs where the constraints are deterministic (for a ﬁxed input dimension), i.e., do not depend on the data X , which is different from the LPs in §2.2–2.4.
Revised: (2019) have considered LPs where the constraints are deterministic (for a ﬁxed input dimension), i.e., do not depend on the data X , which is different from the LPs in §2.2–2.4.

Original: Note that such perturbation techniques in Berthet et al.
Revised: Note that such perturbation techniques in Berthet et al.

Original: (2020) are applicable to our LPs as well.
Revised: (2020) are applicable to our LPs as well.

Original: For example, the Fenchel Young losses as deﬁned in Berthet et al.
Revised: For example, the Fenchel Young losses as deﬁned in Berthet et al.

Original: (2020) is attractive because there is no need to compute the Jacobian.
Revised: (2020) is attractive because there is no need to compute the Jacobian.

Original: From the implementation standpoint, one could simply think of the backward pass as a function given the input and output of the forward pass.
Revised: From the implementation standpoint, one could simply think of the backward pass as a function given the input and output of the forward pass.

Original: But the gradient expressions of the losses is an expectation and hence may require multiple calls to a LP solver in order to approximate the expectation.
Revised: But the gradient expressions of the losses is an expectation and hence may require multiple calls to a LP solver in order to approximate the expectation.

Original: although parallelization and warm starts were shown to alleviate this dependency by sampling in parallel.
Revised: although parallelization and warm starts were shown to alleviate this dependency by sampling in parallel.

Original: Rationale.
Revised: Rationale.

Original: Consider a LP with a large m number of constraints in ﬁxed dimensions n ( n (cid:28) m ).
Revised: Consider a LP with a large m number of constraints in ﬁxed dimensions n ( n (cid:28) m ).

Original: This assumption holds in all formulations in §2.
Revised: This assumption holds in all formulations in §2.

Original: This is because we assume that the architecture is ﬁxed whereas minibatch size depends on the complexity of the task (stable gradient or when noise in gradient is high).
Revised: This is because we assume that the architecture is ﬁxed whereas minibatch size depends on the complexity of the task (stable gradient or when noise in gradient is high).

Original: Hence, solving such LPs using off-the-shelf solvers as in Berthet et al.
Revised: Hence, solving such LPs using off-the-shelf solvers as in Berthet et al.

Original: (2020) may slow down the training process.
Revised: (2020) may slow down the training process.

Original: The strategy in Agrawal et al.
Revised: The strategy in Agrawal et al.

Original: (2019) does offer beneﬁts over
Revised: (2019) does offer beneﬁts over

Original: Amos & Kolter (2017) for sparse QPs.
Revised: Amos & Kolter (2017) for sparse QPs.

Original: Our strategy is to run Mangasarian’s Newton’s method on an exterior penalty function of the LP.
Revised: Our strategy is to run Mangasarian’s Newton’s method on an exterior penalty function of the LP.

Original: There are two advantages: (i) during forward pass, quadratic local convergence of Newton’s method indicates that unrolling the method may be a reasonable choice; and further (ii) based on the relationship between dual and primal variables, and the exactness of the exterior penalty we can show that backward pass is independent of m .
Revised: There are two advantages: (i) during forward pass, quadratic local convergence of Newton’s method indicates that unrolling the method may be a reasonable choice; and further (ii) based on the relationship between dual and primal variables, and the exactness of the exterior penalty we can show that backward pass is independent of m .

Original: We will discuss both these results and some modiﬁcations to deal with discontinuous Hessian (and its inverse) that is required for Newton’s method.
Revised: We will discuss both these results and some modiﬁcations to deal with discontinuous Hessian (and its inverse) that is required for Newton’s method.

Original: A similar approach is proposed in Amos & Kolter (2017) in which
Revised: A similar approach is proposed in Amos & Kolter (2017) in which

Original: Primal-Dual Interior Point methods with implicit differentiation is used for differentiation purposes.
Revised: Primal-Dual Interior Point methods with implicit differentiation is used for differentiation purposes.

Original: But the exterior penalty in (6) satisﬁes an interesting property: primal and dual solutions are related by a closed form expression which can be exploited for efﬁcient backpropagation.
Revised: But the exterior penalty in (6) satisﬁes an interesting property: primal and dual solutions are related by a closed form expression which can be exploited for efﬁcient backpropagation.

Original: 3. F ORWARD P ASS USING N EWTON ’ S A LGORITHM
Revised: 3. F ORWARD P ASS USING N EWTON ’ S A LGORITHM

Original: A key requirement for fast automatic (forward or reverse mode) differentiation is that we can perform the forward pass efﬁciently.
Revised: A key requirement for fast automatic (forward or reverse mode) differentiation is that we can perform the forward pass efﬁciently.

Original: In our setting, we seek to solve and backpropagate through an LP.
Revised: In our setting, we seek to solve and backpropagate through an LP.

Original: We will focus on reverse mode differentiation since it is the most suitable for DNN training.
Revised: We will focus on reverse mode differentiation since it is the most suitable for DNN training.

Original: Given a primal LP, for a ﬁxed accuracy (cid:15) > 0 , Mangasarian (2004) solves an unconstrained problem,
Revised: Given a primal LP, for a ﬁxed accuracy (cid:15) > 0 , Mangasarian (2004) solves an unconstrained problem,

Original: [Equation]
Revised: [Equation]

Original: where σ ( · ) = max( · , 0) represents the elementwise relu function.
Revised: where σ ( · ) = max( · , 0) represents the elementwise relu function.

Original: A modiﬁed Newton’s method can be use to solve equation 6 that performs the following iterations:
Revised: A modiﬁed Newton’s method can be use to solve equation 6 that performs the following iterations:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: In large scale settings of A, b , such Newton methods are known to perform empirically better than gradient descent Mangasarian (2006); Keerthi et al.
Revised: In large scale settings of A, b , such Newton methods are known to perform empirically better than gradient descent Mangasarian (2006); Keerthi et al.

Original: We will discuss if this holds for our purposes shortly.
Revised: We will discuss if this holds for our purposes shortly.

Original: Why is Newton’s method applicable for minibatches?
Revised: Why is Newton’s method applicable for minibatches?

Original: In general, the convergence of Newton’s method depends strongly on initialization, i.e., we can only provide local convergence results.
Revised: In general, the convergence of Newton’s method depends strongly on initialization, i.e., we can only provide local convergence results.

Original: However, this is not the case for our problems since in our examples, either the level sets are bounded from below; or the feasible set is compact, as noted in Mangasarian (2004).
Revised: However, this is not the case for our problems since in our examples, either the level sets are bounded from below; or the feasible set is compact, as noted in Mangasarian (2004).

Original: There are two reasons why the result, by itself, is insufﬁcient for our purposes: (i) it assumes that we can perform line search to satisfy Armijo condition; (ii) even with line search, the result does not provide a rate of convergence.
Revised: There are two reasons why the result, by itself, is insufﬁcient for our purposes: (i) it assumes that we can perform line search to satisfy Armijo condition; (ii) even with line search, the result does not provide a rate of convergence.

Original: In DNN training, such line search based convergence results can be prohibitively expensive.
Revised: In DNN training, such line search based convergence results can be prohibitively expensive.

Original: The main difﬁculty is handling the discontinuity in the Hessian.
Revised: The main difﬁculty is handling the discontinuity in the Hessian.

Original: As a remedy, we use self concordance of
Revised: As a remedy, we use self concordance of

Original: (6) to guarantee global convergence of (7) iterations for the exterior penalty formulation in (6).
Revised: (6) to guarantee global convergence of (7) iterations for the exterior penalty formulation in (6).

Original: To do so, we ﬁrst show a result (proof in Appendix) that characterizes the discrepancy between the actual
Revised: To do so, we ﬁrst show a result (proof in Appendix) that characterizes the discrepancy between the actual

Original: Hessian of (6) and the modiﬁed one in (7) when A is randomly distributed.
Revised: Hessian of (6) and the modiﬁed one in (7) when A is randomly distributed.

Original: Lemma 1.
Revised: Lemma 1.

Original: Assume that A is a random matrix, and ﬁx some y ∈ R n .
Revised: Assume that A is a random matrix, and ﬁx some y ∈ R n .

Original: Then with probability one, g in equation 6 is quadratic (given by ˜ H, ∇ g ( y ) ) over a sufﬁciently small neighborhood of y .
Revised: Then with probability one, g in equation 6 is constant (given by ˜ H, ∇ g ( y ) ) over a sufﬁciently small neighborhood of y .

Original: Intuitively, Lemma 1 states that with probability one, each y has a neighborhood in which the Hessian is quadratic.
Revised: Intuitively, Lemma 1 states that with probability one, each y has a neighborhood in which the Hessian is constant.

Original: In addition, the modiﬁed Hessian is nonsingular at all points (in particular the optimal y ∗ ), and so we can then show the following global convergence result.
Revised: In addition, the modiﬁed Hessian is nonsingular at all points (in particular the optimal y ∗ ), and so we can then show the following global convergence result.

Original: Theorem 2.
Revised: Theorem 2.

Original: Newton’s method converges globally at a linear rate with local quadratic convergence.
Revised: Newton’s method converges globally at a linear rate with local quadratic convergence.

Original: Proof.
Revised: Proof.

Original: (Sketch)
Revised: (Sketch)

Original: First, we use the fact that the objective in equation 6 is piecewise quadratic, and hence self concordant.
Revised: First, we use the fact that the objective in equation 6 is piecewise quadratic, and hence self concordant.

Original: Second, observe that the possible choice of Hessian are ﬁnite, and so we can choose ρ > 0 so that there is a descent direction without line search, that is, there exist a step size λ > 0 such that λ ∇ g ( x ) T d < 0 .
Revised: Second, observe that the possible choice of Hessian are ﬁnite, and so we can choose ρ > 0 so that there is a descent direction without line search, that is, there exist a step size λ > 0 such that λ ∇ g ( x ) T d < 0 .

Original: Finally, we use Theorem 4.1.12 in Nesterov (2013) to claim the result.
Revised: Finally, we use Theorem 4.1.12 in Nesterov (2013) to claim the result.

Original: Please see Appendix for the proof.
Revised: Please see Appendix for the proof.

Original: The loss function is almost surely quadratic in the neighborhood of any y , thus intuitively suggests local quadratic convergence independent of the starting point.
Revised: The loss function is almost surely quadratic in the neighborhood of any y , thus intuitively suggests local quadratic convergence independent of the starting point.

Original: Remark 3.
Revised: Remark 3.

Original: Convergence in Thm.
Revised: Convergence in Thm.

Original: 2 is guaranteed under standard constraint qualiﬁcation assumptions.
Revised: 2 is guaranteed under standard constraint qualiﬁcation assumptions.

Original: Linear Independent Constraint Qualiﬁcation (LICQ) is satisﬁed for AUC, and Multi-class AUC formulations in §2.
Revised: Linear Independent Constraint Qualiﬁcation (LICQ) is satisﬁed for AUC, and Multi-class AUC formulations in §2.

Original: But the F -score formulation does not satisfy LICQ, hence we need safeguarding principles in the initial iterations (until iterates get close to the optimal solution).
Revised: But the F -score formulation does not satisfy LICQ, hence we need safeguarding principles in the initial iterations (until iterates get close to the optimal solution).

Original: 3. B ACKWARD P ASS USING O PTIMAL D UAL V ARIABLES A IDED BY U NROLLING
Revised: 3. B ACKWARD P ASS USING O PTIMAL D UAL V ARIABLES A IDED BY U NROLLING

Original: The advantage of optimizing the exterior penalty in (6) is that given an iterate y t , accuracy (cid:15) , we can extract the optimal dual solution v t by simple thresholding, that is, v t = 1 /(cid:15)
Revised: The advantage of optimizing the exterior penalty in (6) is that given an iterate y t , accuracy (cid:15) , we can extract the optimal dual solution v t by simple thresholding, that is, v t = 1 /(cid:15)

Original: ( A T σ ( Ay − b )) .
Revised: ( A T σ ( Ay − b )) .

Original: By complementarity slackness, the nonzero coordinates of v t specify the set of active constraints in
Revised: By complementarity slackness, the nonzero coordinates of v t specify the set of active constraints in

Original: Ax ≤ b .
Revised: Ax ≤ b .

Original: So, given an approximate solution y t such that ∇ g ( y t ) ˜ H ( y t ) − 1 ∇ g ( y t ) ≤ (cid:15) , to compute the primal solution x ∗ , we solve the “active" linear system given by ˜ A ˜ b , where ˜ A denotes the active rows of A and the corresponding subvector ˜ b .
Revised: So, given an approximate solution y t such that ∇ g ( y t ) ˜ H ( y t ) − 1 ∇ g ( y t ) ≤ (cid:15) , to compute the primal solution x ∗ , we solve the “active" linear system given by ˜ A ˜ b , where ˜ A denotes the active rows of A and the corresponding subvector ˜ b .

Original: Hence, backpropagation through the layer reduces to computing derivatives of ˜ A − 1 ˜ b which is simple via automatic differentiation.
Revised: Hence, backpropagation through the layer reduces to computing derivatives of ˜ A − 1 ˜ b which is simple via automatic differentiation.

Original: How to choose (cid:15) ?
Revised: How to choose (cid:15) ?

Original: If we can successfully retrieve the active constraints at the optimal solution, we do not need to store the intermediate iterates y t at all during the forward pass (memory efﬁcient).
Revised: If we can successfully retrieve the active constraints at the optimal solution, we do not need to store the intermediate iterates y t at all during the forward pass (memory efﬁcient).

Original: However, setting (cid:15) correctly can be tricky for arbitrary polyhedra since it depends on the geometric properties such as facets and vertices that may be difﬁcult to enumerate.
Revised: However, setting (cid:15) correctly can be tricky for arbitrary polyhedra since it depends on the geometric properties such as facets and vertices that may be difﬁcult to enumerate.

Original: One possible way to get around this is to use a “burn-in” period in which we increase (cid:15) slightly in each iteration (of deep network training) and backpropagate through the unrolled Newton’s iterations during this period.
Revised: One possible way to get around this is to use a “burn-in” period in which we increase (cid:15) slightly in each iteration (of deep network training) and backpropagate through the unrolled Newton’s iterations during this period.

Original: Once we see that the convergence proﬁle has stabilized, we can ﬁx (cid:15) at that value and start using the complementarity conditions and derive the active linear system ˜
Revised: Once we see that the convergence proﬁle has stabilized, we can ﬁx (cid:15) at that value and start using the complementarity conditions and derive the active linear system ˜

Original: A − 1 ˜ b as discussed above.
Revised: A − 1 ˜ b as discussed above.

Original: How to backpropagate through unrolled iterations?
Revised: How to backpropagate through unrolled iterations?

Original: We assume that the chain rule is applicable up to this LP layer and is ∂L∂x ∂x∂A (for one of the parameters A ), and note that it is possible to ﬁnd ∂L∂x (either directly or using a chain rule).
Revised: We assume that the chain rule is applicable up to this LP layer and is ∂L∂x ∂x∂A (for one of the parameters A ), and note that it is possible to ﬁnd ∂L∂x (either directly or using a chain rule).

Original: Therefore we focus our attention on ∂x∂A , which involves the LP layer.
Revised: Therefore we focus our attention on ∂x∂A , which involves the LP layer.

Original: Indeed, unrolling each iteration in (7) is equivalent to a “sublayer".
Revised: Indeed, unrolling each iteration in (7) is equivalent to a “sublayer".

Original: So in order to backpropagate we have to show the partial derivatives of each operation or step wrt to the LP parameters A, b, and c .
Revised: So in order to backpropagate we have to show the partial derivatives of each operation or step wrt to the LP parameters A, b, and c .

Original: Our goal is to ∂d∂A where d = Q − 1 q , Q =
Revised: Our goal is to ∂d∂A where d = Q − 1 q , Q =

Original: ˜ H and u = ∇ g
Revised: ˜ H and u = ∇ g

Original: ( y ) .
Revised: ( y ) .

Original: We can use the product rule to arrive at: ∂d = − (cid:0) u T Q − 1 ⊗ Q − 1 (cid:1)
Revised: We can use the product rule to arrive at: ∂d = − (cid:0) u T Q − 1 ⊗ Q − 1 (cid:1)

Original: ∂Q + Q − 1 ∂u .
Revised: ∂Q + Q − 1 ∂u .

Original: To see this, note that we have used the chain rule to differentiate through the inverse in the ﬁrst term.
Revised: To see this, note that we have used the chain rule to differentiate through the inverse in the ﬁrst term.

Original: The second term is easy to compute similar to the computation of Hessian.
Revised: The second term is easy to compute similar to the computation of Hessian.

Original: For each of these terms we eventually have to compute ∂Q∂z or ∂u∂z where z ∈ { c, A, b } which can also be done by another application of chain rule.
Revised: For each of these terms we eventually have to compute ∂Q∂z or ∂u∂z where z ∈ { c, A, b } which can also be done by another application of chain rule.

Original: Please see Appendix A. for empirical veriﬁcation of unrolled gradient and the one provided by ˜ A − 1 ˜ b .
Revised: Please see Appendix A. for empirical veriﬁcation of unrolled gradient and the one provided by ˜ A − 1 ˜ b .

Original: Before proceeding, we should note an issue that comes up when differentiating each step of the unrolled algorithm due to the fact that the Hessian is piecewise linear (constant) as a function of the input to that particular layer.
Revised: Before proceeding, we should note an issue that comes up when differentiating each step of the unrolled algorithm due to the fact that the Hessian is piecewise linear (constant) as a function of the input to that particular layer.

Original: Here, some possible numerical approximations are needed, as we describe below.
Revised: Here, some possible numerical approximations are needed, as we describe below.

Original: Remark 4.
Revised: Remark 4.

Original: Note that the diagonal matrix term in ∂Q∂A is nondifferentiable due to the presence of the step function.
Revised: Note that the diagonal matrix term in ∂Q∂A is nondifferentiable due to the presence of the step function.

Original: However, the step function is a piecewise constant function, and hence has zero derivative almost surely, that is, in any bounded set S , x ∈ S , if a ball (of radius r > 0 ,)
Revised: However, the step function is a piecewise constant function, and hence has zero derivative almost surely, that is, in any bounded set S , x ∈ S , if a ball (of radius r > 0 ,)

Original: B r ( x ) ⊆ S , then the Lebesgue measure of the set of nondifferentiable points on S is zero.
Revised: B r ( x ) ⊆ S , then the Lebesgue measure of the set of nondifferentiable points on S is zero.

Original: Please see Appendix
Revised: Please see Appendix

Original: B.3 for a formal justiﬁcation where we show this by approximating the step function using a sequence of logistic functions with increasing slope parameter at the origin.
Revised: B.3 for a formal justiﬁcation where we show this by approximating the step function using a sequence of logistic functions with increasing slope parameter at the origin.

Original: Therefore, in this setting, Remark 4 provides a way to compute an approximate sub-gradient when using Newton’s method based LP layers.
Revised: Therefore, in this setting, Remark 4 provides a way to compute an approximate sub-gradient when using Newton’s method based LP layers.

Original: The function is a piece-wise quadratic function and differentiable everywhere, and the inverse of the Hessian acts as a preconditioner.
Revised: The function is a piece-wise quadratic function and differentiable everywhere, and the inverse of the Hessian acts as a preconditioner.

Original: Summary .
Revised: Summary .

Original: Our forward pass involved three steps: 1.
Revised: Our forward pass involved three steps: 1.

Original: ﬁnite steps of Newton’s method using which we 2.
Revised: ﬁnite steps of Newton’s method using which we 2.

Original: computed the dual variable by a thresholding operation, and 3.
Revised: computed the dual variable by a thresholding operation, and 3.

Original: ﬁnally, to get the primal solution, these dual variables are ﬁrst used to identify the active constraints followed by solving a linear system.
Revised: ﬁnally, to get the primal solution, these dual variables are ﬁrst used to identify the active constraints followed by solving a linear system.

Original: In order to backpropagate through these three steps, we must differentiate through each layer of our procedure including ˜ A − 1 ˜ b , independent of whether we use unrolling or Danskin’s theorem.
Revised: In order to backpropagate through these three steps, we must differentiate through each layer of our procedure including ˜ A − 1 ˜ b , independent of whether we use unrolling or Danskin’s theorem.

Original: Using Danskin’s theorem in this setting would involve differentiating through the ﬁxed point of the
Revised: Using Danskin’s theorem in this setting would involve differentiating through the ﬁxed point of the

Original: Newton’s iterations similar to (regularized) gradient descent iterations considered in the iMAML work Rajeswaran et al.
Revised: Newton’s iterations similar to (regularized) gradient descent iterations considered in the iMAML work Rajeswaran et al.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we conduct experiments on commonly used benchmarks to show that our framework can be used to optimize multiple different objectives within deep neural networks and lead to performance gain.
Revised: In this section, we conduct experiments on commonly used benchmarks to show that our framework can be used to optimize multiple different objectives within deep neural networks and lead to performance gain.

Original: We start with binary AUC optimization, and then extend to multi class AUC optimization and F -score optimization.
Revised: We start with binary AUC optimization, and then extend to multi class AUC optimization and F -score optimization.

Original: We also show that nonnegative matrix factorization can be optimized in linear programming form in our framework.
Revised: We also show that nonnegative matrix factorization can be optimized in linear programming form in our framework.

Original: Optimizing
Revised: Optimizing

Original: Binary AUC
Revised: Binary AUC

Original: We follow the current state-of-the-art work on AUC optimization Liu et al.
Revised: We follow the current state-of-the-art work on AUC optimization Liu et al.

Original: (2019) to conduct experiments on optimizing AUC score directly with deep neural networks.
Revised: (2019) to conduct experiments on optimizing AUC score directly with deep neural networks.

Original: The baseline algorithms we compare with for binary AUC are cross-entropy loss and two algorithm (PPD-SG and PPDAdaGrad) from Liu et al.
Revised: The baseline algorithms we compare with for binary AUC are cross-entropy loss and two algorithm (PPD-SG and PPDAdaGrad) from Liu et al.

Original: Datasets: Cat&Dog, CIFAR10, CIFAR100, and STL10.
Revised: Datasets: Cat&Dog, CIFAR10, CIFAR100, and STL10.

Original: Cat&Dog is a dataset from Kaggle which contains 25000 images of cats and dogs.
Revised: Cat&Dog is a dataset from Kaggle which contains 25000 images of cats and dogs.

Original: 80% of the dataset is used as training set and the rest 20% as test set.
Revised: 80% of the dataset is used as training set and the rest 20% as test set.

Original: STL10 is inspired by the CIFAR-10 dataset but with some modiﬁcations.
Revised: STL10 is inspired by the CIFAR-10 dataset but with some modiﬁcations.

Original: Each class in
Revised: Each class in

Original: STL10 has fewer labeled training examples than in CIFAR-10.
Revised: STL10 has fewer labeled training examples than in CIFAR-10.

Original: We follow Liu et al.
Revised: We follow Liu et al.

Original: (2019) to use 19k/1k, 45k/5k, 45k/5k, 4k/1k training/validation split on Cat&Dog, CIFAR10, CIFAR100, STL respectively.
Revised: (2019) to use 19k/1k, 45k/5k, 45k/5k, 4k/1k training/validation split on Cat&Dog, CIFAR10, CIFAR100, STL respectively.

Original: [Table]
Revised: [Table]

Original: 8 PPD-SG 56 .

Revised: 8 PPD-SG 56 .

Original: 4 PPD-AdaGrad 56 .

Revised: 4 PPD-AdaGrad 56 .

Original: 5 Ours 58.
Revised: 5 Ours 58.

Original: Construction of imbalanced datasets:
Revised: Construction of imbalanced datasets:

Original: We construct imbalanced binary classiﬁcation task by using half classes as positive class and another half as negative class, and dropping samples from negative class by a certain ratio, which is reﬂected by the positive ratio (the ratio of the majority class to the minority class) in Table 1.
Revised: We construct imbalanced binary classiﬁcation task by using half classes as positive class and another half as negative class, and dropping samples from negative class by a certain ratio, which is reﬂected by the positive ratio (the ratio of the majority class to the minority class) in Table 1.

Original: Experimental setting.
Revised: Experimental setting.

Original: We use a Resnet- 18 He et al.
Revised: We use a Resnet- 18 He et al.

Original: as the deep neural network for all algorithms.
Revised: as the deep neural network for all algorithms.

Original: During optimization, the batch size is set to 64 .
Revised: During optimization, the batch size is set to 64 .

Original: The initial learning rate is tuned in { 0 . 1 , 0 . 01 , 0 . 001 } and decays 2 / 3 at 2 k, 10 k, 25 k -th iteration.
Revised: The initial learning rate is tuned in { 0 . 1 , 0 . 01 , 0 . 001 } and decays 2 / 3 at 2 k, 10 k, 25 k -th iteration.

Original: We train 40 k iterations in total.
Revised: We train 40 k iterations in total.

Original: The (cid:15) in
Revised: The (cid:15) in

Original: Newton’s method is 0 .
Revised: Newton’s method is 0 .

Original: We use the same random seed, learning rate and total number of iterations in all of our experiments including multi class AUC and F -score experiments.
Revised: We use the same random seed, learning rate and total number of iterations in all of our experiments including multi class AUC and F -score experiments.

Original: [Table]
Revised: [Table]

Original: Results.
Revised: Results.

Original: The results are shown in Table 1.
Revised: The results are shown in Table 1.

Original: We can see that our method slightly outperforms
Revised: We can see that our method slightly outperforms

Original: Liu et al.
Revised: Liu et al.

Original: and outperforms cross-entropy loss by a large margin, especially on imbalanced datasets, where AUC objective shows superiority over cross-entropy loss.
Revised: and outperforms cross-entropy loss by a large margin, especially on imbalanced datasets, where AUC objective shows superiority over cross-entropy loss.

Original: Inﬂuence of (cid:15) .
Revised: Inﬂuence of (cid:15) .

Original: We report the inﬂuence of (cid:15) on our algorithm in Table 2, where we choose Cat&Dog as an example to test different (cid:15) .
Revised: We report the inﬂuence of (cid:15) on our algorithm in Table 2, where we choose Cat&Dog as an example to test different (cid:15) .

Original: We can see that (cid:15) = 0 .
Revised: We can see that (cid:15) = 0 .

Original: 1 gets slightly worse performance than (cid:15)
Revised: 1 gets slightly worse performance than (cid:15)

Original: = 0 .
Revised: = 0 .

Original: 01 , while (cid:15) = 0 .
Revised: 01 , while (cid:15) = 0 .

Original: 001 performs much worse.
Revised: 001 performs much worse.

Original: To choose (cid:15) , we follow the approach proposed by
Revised: To choose (cid:15) , we follow the approach proposed by

Original: Mangasarian (2004).
Revised: Mangasarian (2004).

Original: If for two successive values of (cid:15) 1 > (cid:15) 2 , the value of the (cid:15) perturbed quadratic function is the same, then it is the least 2-norm solution of the dual.
Revised: If for two successive values of (cid:15) 1 > (cid:15) 2 , the value of the (cid:15) perturbed quadratic function is the same, then it is the least 2-norm solution of the dual.

Original: Therefore, we simply choose an (cid:15) that satisﬁes this property, which is chosen to be 0 .
Revised: Therefore, we simply choose an (cid:15) that satisﬁes this property, which is chosen to be 0 .

Original: 01 in our experiments.
Revised: 01 in our experiments.

Original: [Table]
Revised: [Table]

Original: Optimizing Multiclass AUC
Revised: Optimizing Multiclass AUC

Original: We further demonstrate our method for optimizing multiclass AUC.
Revised: We further demonstrate our method for optimizing multiclass AUC.

Original: Similar to previous section on binary
Revised: Similar to previous section on binary

Original: AUC, we construct imbalanced multiclass datasets by dividing datasets intoclasses and drop samples from 2 of them and report the one-versus-all AUC (denoted as AUC ova ) and AUC bin µ score .
Revised: AUC, we construct imbalanced multiclass datasets by dividing datasets intoclasses and drop samples from 2 of them and report the one-versus-all AUC (denoted as AUC ova ) and AUC bin µ score .

Original: For STL10, we group class 0 − 2 , 3 − 5 , 6 − 9 into the three classes, and drop samples from the ﬁrst two classes.
Revised: For STL10, we group class 0 − 2 , 3 − 5 , 6 − 9 into the three classes, and drop samples from the ﬁrst two classes.

Original: For CIFR100, we group class 0 − 32 , 33 − 65 , 66 − 99 into three classes, and also drop samples from the ﬁrst two classes.
Revised: For CIFR100, we group class 0 − 32 , 33 − 65 , 66 − 99 into three classes, and also drop samples from the ﬁrst two classes.

Original: Results.
Revised: Results.

Original: Results are in Table 3.
Revised: Results are in Table 3.

Original: In addition to one-versus-all AUC metric, we also report the performance in terms of AUC µ Kleiman & Page (2019) which is speciﬁcally designed for measuring multiclass AUC and keeps nice properties of binary AUC such as being insensitive to class skew.
Revised: In addition to one-versus-all AUC metric, we also report the performance in terms of AUC µ Kleiman & Page (2019) which is speciﬁcally designed for measuring multiclass AUC and keeps nice properties of binary AUC such as being insensitive to class skew.

Original: We can see that our method outperforms cross-entropy loss on all four datasets and under all different skewed ratios.
Revised: We can see that our method outperforms cross-entropy loss on all four datasets and under all different skewed ratios.

Original: Speciﬁcally, the performance gain tends to be larger when the dataset becomes more imbalanced.
Revised: Speciﬁcally, the performance gain tends to be larger when the dataset becomes more imbalanced.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: We show that by directly optimizing F -score, we can achieve a better performance on this than when using cross entropy loss.
Revised: We show that by directly optimizing F -score, we can achieve a better performance on this than when using cross entropy loss.

Original: In addition to cross entropy loss, we perform evalua- tions with two other methods that can also directly optimize the F 1 -score.
Revised: In addition to cross entropy loss, we perform evalua- tions with two other methods that can also directly optimize the F 1 -score.

Original: First, we replace our solver with CVXPY-SCS Agrawal et al.
Revised: First, we replace our solver with CVXPY-SCS Agrawal et al.

Original: (2019), which is a differentiable general purpose linear programming solver
Revised: (2019), which is a differentiable general purpose linear programming solver

Original: ; second, we perform comparisons with AP-Perf Fathony & Kolter (2020) which offers differentiable optimization of F -score using an adversarial prediction framework.
Revised: ; second, we perform comparisons with AP-Perf Fathony & Kolter (2020) which offers differentiable optimization of F -score using an adversarial prediction framework.

Original: The datasets and our setup to group them into two classes remain the same as in binary AUC section.
Revised: The datasets and our setup to group them into two classes remain the same as in binary AUC section.

Original: The results in Table 4 shows that our method consistent improvement over cross-entropy loss in terms of F -score.
Revised: The results in Table 4 shows that our method generally yields improvement over cross-entropy loss in terms of F -score.

Original: Note that different from optimizing cross entropy loss, when we optimize F -score directly, there exists a local optimal point where assigning all examples to the positive class leads to F -score of 66 .
Revised: Note that different from optimizing cross entropy loss, when we optimize F -score directly, there exists a local optimal point where assigning all examples to the positive class leads to F -score of 66 .

Original: We see this behavior on CIFAR10, CIFAR100, and STL10.
Revised: We see this behavior on CIFAR10, CIFAR100, and STL10.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We demonstrated that various non-decomposable objectives can be optimized within deep neural networks in a differentiable way under the same general framework of LPs using a modiﬁed Newton’s algorithm proposed by Mangasarian.
Revised: We demonstrated that various non-decomposable objectives can be optimized within deep neural networks in a differentiable way under the same general framework of LPs using a modiﬁed Newton’s algorithm proposed by Mangasarian.

Original: A number of recent papers have studied the general problem of backpropagating through convex optimization modules, and this literature provides several effective approaches although scalability remains a topic of active research.
Revised: A number of recent papers have studied the general problem of backpropagating through convex optimization modules, and this literature provides several effective approaches although scalability remains a topic of active research.

Original: Our work complements these results and shows that the operations needed can be implemented to utilize the capabilities of modern deep learning libraries.
Revised: Our work complements these results and shows that the operations needed can be implemented to utilize the capabilities of modern deep learning libraries.

Original: While our experimental results suggest that promising results on binary AUC, multi-class AUC and F -score optimization within DNNs is achievable, we believe that the module may have other applications where the number of constraints are large and data-dependent.
Revised: While our experimental results suggest that promising results on binary AUC, multi-class AUC and F -score optimization within DNNs is achievable, we believe that the module may have other applications where the number of constraints are large and data-dependent.

Original: A A PPENDIX A. LP FORMULATION FOR MULTI - CLASS AUC
Revised: A A PPENDIX A. LP FORMULATION FOR MULTI - CLASS AUC

Original: One way to extend binary AUC to multi-class is by considering multiple one-versus-all pairs.
Revised: One way to extend binary AUC to multi-class is by considering multiple one-versus-all pairs.

Original: This leads us to the following formulation:
Revised: This leads us to the following formulation:

Original: [Equation]
Revised: [Equation]

Original: In our multi-class AUC experiment, we use this one-versus-all AUC as training loss and report performance in both one-versus-all AUC and AUC bin µ .
Revised: In our multi-class AUC experiment, we use this one-versus-all AUC as training loss and report performance in both one-versus-all AUC and AUC bin µ .

Original: In addition, we can also consider the setting of
Revised: In addition, we can also consider the setting of

Original: AUC µ where P is set arbitarily.
Revised: AUC µ where P is set arbitarily.

Original: In this case, the exact terms in orientation function O proposed by
Revised: In this case, the exact terms in orientation function O proposed by

Original: Kleiman & Page (2019) can be written as follows:
Revised: Kleiman & Page (2019) can be written as follows:

Original: [Equation]
Revised: [Equation]

Original: Note that AUC a rbit µ has the same number of constraints and variables as AUC b in µ .
Revised: Note that AUC a rbit µ has the same number of constraints and variables as AUC b in µ .

Original: Once the LPs are solved, the loss function is calculated the same way as binary AUC.
Revised: Once the LPs are solved, the loss function is calculated the same way as binary AUC.

Original: A. F ORMULATING R ATIO O BJECTIVES
Revised: A. F ORMULATING R ATIO O BJECTIVES

Original: In this section, we study a subset of non-decomposable metrics, which are typically expressed as ratios of some combination of True Positive(TP), False Positives(FP), True Negatives(TN) and False
Revised: In this section, we study a subset of non-decomposable metrics, which are typically expressed as ratios of some combination of True Positive(TP), False Positives(FP), True Negatives(TN) and False

Original: Negatives(FN).
Revised: Negatives(FN).

Original: These can be expressed in a general form as a 11 TP + a 12 a 21 TP + a 22 FP + a 23 FN + a 24 , where a pq are constants/cofﬁcients which if set to 0, means the term is absent and not equal to zero in other cases.
Revised: These can be expressed in a general form as a 11 TP + a 12 a 21 TP + a 22 FP + a 23 FN + a 24 , where a pq are constants/cofﬁcients which if set to 0, means the term is absent and not equal to zero in other cases.

Original: This formulation can used to deﬁne Fscore, F β , Jaccard, IOU and Precision at ﬁxed recall.
Revised: This formulation can used to deﬁne Fscore, F β , Jaccard, IOU and Precision at ﬁxed recall.

Original: In the following section, we describe the formulation of Fscore as a representative of this approach, other metrics can be formulated similarly.
Revised: In the following section, we describe the formulation of Fscore as a representative of this approach, other metrics can be formulated similarly.

Original: Given Y the groud truth, our goal is to compute ˆ Y both of length n , which aligns with Y based on the speciﬁc metric.
Revised: Given Y the groud truth, our goal is to compute ˆ Y both of length n , which aligns with Y based on the speciﬁc metric.

Original: We ﬁrst show how to write TP, FP, TN and FN wrt to these vectors.
Revised: We ﬁrst show how to write TP, FP, TN and FN wrt to these vectors.

Original: [Equation]
Revised: [Equation]

Original: A.2. F ORMULATING F - SCORE
Revised: A.2. F ORMULATING F - SCORE

Original: The F -score or F -measure is routinely used as a performance metric for different types of prediction problems, including binary classiﬁcation and, multi-label classiﬁcation.
Revised: The F -score or F -measure is routinely used as a performance metric for different types of prediction problems, including binary classiﬁcation and, multi-label classiﬁcation.

Original: Compared to measures like error rate in binary classiﬁcation and Hamming loss, it enforces a better balance between performance on the minority and the majority classes, and, therefore, it is more suitable in the case of imbalanced data Dembczynski et al.
Revised: Compared to measures like error rate in binary classiﬁcation and Hamming loss, it enforces a better balance between performance on the minority and the majority classes, and, therefore, it is more suitable in the case of imbalanced data Dembczynski et al.

Original: F -score is deﬁned as follows:
Revised: F -score is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: where P is the measure of precision deﬁned as
Revised: where P is the measure of precision deﬁned as

Original: [Equation]
Revised: [Equation]

Original: and R stands for the measure of recall, given as
Revised: and R stands for the measure of recall, given as

Original: [Equation]
Revised: [Equation]

Original: Plugging this in Eq(1), and replacing the formulations for TP, FP and FN(from Eq A.2) we get
Revised: Plugging this in Eq(1), and replacing the formulations for TP, FP and FN(from Eq A.2) we get

Original: F - score ( Y,
Revised: F - score ( Y,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where y i refers to the ith element of Y (same for ˆ y i ).
Revised: where y i refers to the ith element of Y (same for ˆ y i ).

Original: 1 represents an all one vector in R n .
Revised: 1 represents an all one vector in R n .

Original: Note that in training, since Y is generally provided, we can assume 1 T Y = β which is constant (the number of examples in the positive class in the ground truth).
Revised: Note that in training, since Y is generally provided, we can assume 1 T Y = β which is constant (the number of examples in the positive class in the ground truth).

Original: We can also represent the the values of 2 Y as a coefﬁcient matrix c , then the optimization problem for ﬁnding F -score can be written as
Revised: We can also represent the the values of 2 Y as a coefﬁcient matrix c , then the optimization problem for ﬁnding F -score can be written as

Original: [Equation]
Revised: [Equation]

Original: subject to
Revised: subject to

Original: [Equation]
Revised: [Equation]

Original: M AXIMIZING J ACCARD C OEFFICIENT AND F β
Revised: M AXIMIZING J ACCARD C OEFFICIENT AND F β

Original: The Jaccard Coefﬁcient and Dice Index lead to similar formulation as F -score.
Revised: The Jaccard Coefﬁcient and Dice Index lead to similar formulation as F -score.

Original: The Jaccard coefﬁcient can be expressed as:
Revised: The Jaccard coefﬁcient can be expressed as:

Original: [Equation]
Revised: [Equation]

Original: This can be equivalently written as a linear factional program as shown in Model(10) where c =
Revised: This can be equivalently written as a linear factional program as shown in Model(10) where c =

Original: Y , d
Revised: Y , d

Original: = (1 − Y ) and b = 1 T Y .
Revised: = (1 − Y ) and b = 1 T Y .

Original: The rest of the construction is similar to F -score.
Revised: The rest of the construction is similar to F -score.

Original: Note that F β which is deﬁned as
Revised: Note that F β which is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where β is a user speciﬁed parameter (balancing the importance of precision and recall) also permits a similar formulation.
Revised: where β is a user speciﬁed parameter (balancing the importance of precision and recall) also permits a similar formulation.

Original: Here we simply set c = (1 + β 2 ) Y , d = 1 and b = β 2 1 T Y .
Revised: Here we simply set c = (1 + β 2 ) Y , d = 1 and b = β 2 1 T Y .

Original: A.2. M AXIMIZING P @ R
Revised: A.2. M AXIMIZING P @ R

Original: We begin by deﬁning the maximum precision at ﬁxed minimum recall problem as
Revised: We begin by deﬁning the maximum precision at ﬁxed minimum recall problem as

Original: [Equation]
Revised: [Equation]

Original: This is again a linear fractional objective with a linear constraint.
Revised: This is again a linear fractional objective with a linear constraint.

Original: So we can write it as an equivalent
Revised: So we can write it as an equivalent

Original: Linear program using the same transformation
Revised: Linear program using the same transformation

Original: where c =
Revised: where c =

Original: Y , d = 1 and b = 0 .
Revised: Y , d = 1 and b = 0 .

Original: R @
Revised: R @

Original: P on the other hand directly leads to a linear program.
Revised: P on the other hand directly leads to a linear program.

Original: A. O PTIMIZING N ON - NEGATIVE M ATRIX F ACTORIZATION (NMF)
Revised: A. O PTIMIZING N ON - NEGATIVE M ATRIX F ACTORIZATION (NMF)

Original: Nonnegative matrix factorization is different from the other objectives we presented in that it is primarily used in unsupervised learning and does not satify the criteria for a metric loss function.
Revised: Nonnegative matrix factorization is different from the other objectives we presented in that it is primarily used in unsupervised learning and does not satify the criteria for a metric loss function.

Original: It can still be formulated in a generalized non-decomposable form because (i) cannot be written as a sum over individual samples, (ii) leads to a model where the constraints depend on learned features.
Revised: It can still be formulated in a generalized non-decomposable form because (i) cannot be written as a sum over individual samples, (ii) leads to a model where the constraints depend on learned features.

Original: Note that purpose of discussing NMF in this context, is not to provide a general purpose solver for the problem, and instead to assess whether NMF layers can serve as a regularizer or a clustering module, e.g., learning more interpretable attributes, co-segmentation and a substitute for clustering, see Trigeorgis et al.
Revised: Note that purpose of discussing NMF in this context, is not to provide a general purpose solver for the problem, and instead to assess whether NMF layers can serve as a regularizer or a clustering module, e.g., learning more interpretable attributes, co-segmentation and a substitute for clustering, see Trigeorgis et al.

Original: and Collins et al.
Revised: and Collins et al.

Original: The following description in this section and the experimental validation in the following section is a proof of principle instantiation of this idea.
Revised: The following description in this section and the experimental validation in the following section is a proof of principle instantiation of this idea.

Original: We brieﬂy review from Arora et al.
Revised: We brieﬂy review from Arora et al.

Original: (2012) and Recht et al.
Revised: (2012) and Recht et al.

Original: (2012), how NMF is written as a LP.
Revised: (2012), how NMF is written as a LP.

Original: We know from Arora et al.
Revised: We know from Arora et al.

Original: (2012) that a NMF decomposition V = FW where F is s × s (cid:48) and W is s (cid:48) × w and V and W have a row sum of 1 , is ‘separable’ if the rows of W are simplicial and there is a permutation matrix R ∈ R s × s such that RF =
Revised: (2012) that a NMF decomposition V = FW where F is s × s (cid:48) and W is s (cid:48) × w and V and W have a row sum of 1 , is ‘separable’ if the rows of W are simplicial and there is a permutation matrix R ∈ R s × s such that RF =

Original: [ I r
Revised: [ I r

Original: M ] T .
Revised: M ] T .

Original: The top r rows of F contains so-called anchor words.
Revised: The top r rows of F contains so-called anchor words.

Original: Recht et al.
Revised: Recht et al.

Original: (2012) proposed a data-driven model where the most salient features in the data are used to express the remaining features, given as V ∼ CV , where C is of size s × s .
Revised: (2012) proposed a data-driven model where the most salient features in the data are used to express the remaining features, given as V ∼ CV , where C is of size s × s .

Original: Assuming V admits a rank- r separable factorization, then V = R T (cid:20) I r
Revised: Assuming V admits a rank- r separable factorization, then V = R T (cid:20) I r

Original: M (cid:21)
Revised: M (cid:21)

Original: RV = CV .
Revised: RV = CV .

Original: To show that thsis factorization is possible, we need to ﬁrst make F square, which is why it is zero-padded to make it a size s × s matrix.
Revised: To show that thsis factorization is possible, we need to ﬁrst make F square, which is why it is zero-padded to make it a size s × s matrix.

Original: Let (cid:101) p be any vector which is used as the coefﬁcient in the objective in the following model.
Revised: Let (cid:101) p be any vector which is used as the coefﬁcient in the objective in the following model.

Original: According to Recht et al.
Revised: According to Recht et al.

Original: (2012), any value for the entries of (cid:101) p should sufﬁce as long as they are distinct.
Revised: (2012), any value for the entries of (cid:101) p should sufﬁce as long as they are distinct.

Original: with distinct values.
Revised: with distinct values.

Original: Then the LP formulation is as follows:
Revised: Then the LP formulation is as follows:

Original: [Equation]
Revised: [Equation]

Original: With C in hand, W is constructed by extracting rows of V for those indices k where C kk = 1 .
Revised: With C in hand, W is constructed by extracting rows of V for those indices k where C kk = 1 .

Original: F is constructed by extracting rows of C which correspond to k where C kk = 1 .
Revised: F is constructed by extracting rows of C which correspond to k where C kk = 1 .

Original: A.3.
Revised: A.3.

Original: E XPERIMENTAL RESULTS ON N ONNEGATIVE M ATRIX F ACTORIZATION
Revised: E XPERIMENTAL RESULTS ON N ONNEGATIVE M ATRIX F ACTORIZATION

Original: = 2 respectively.
Revised: = 2 respectively.

Original: We demonstrate applicability of our strategy to nonnegative matrix factorization (NMF) by performing a rank k factorization on
Revised: We demonstrate applicability of our strategy to nonnegative matrix factorization (NMF) by performing a rank k factorization on

Original: Convolutional Neural Network (CNN) activations as an example, following Collins et al.
Revised: Convolutional Neural Network (CNN) activations as an example, following Collins et al.

Original: Recall that the activation tensor of an image at some layer in CNN has the shape V ∈ R c × h × w where h, w are the spatial sizes and c is the number of channels.
Revised: Recall that the activation tensor of an image at some layer in CNN has the shape V ∈ R c × h × w where h, w are the spatial sizes and c is the number of channels.

Original: We can reshape it into V ∈ R c
Revised: We can reshape it into V ∈ R c

Original: ×
Revised: ×

Original: ( h · w ) and calculate a rank k NMF for V :
Revised: ( h · w ) and calculate a rank k NMF for V :

Original: V = FW .
Revised: V = FW .

Original: Each row W j of the resultant W ∈ R k × ( h · w ) can be reshaped into a heat map of dimension h × w which highlights regions in the image that correspond to the factor W j .
Revised: Each row W j of the resultant W ∈ R k × ( h · w ) can be reshaped into a heat map of dimension h × w which highlights regions in the image that correspond to the factor W j .

Original: We show an example for k = 1 , 2 in Fig.
Revised: We show an example for k = 1 , 2 in Fig.

Original: We can see that heatmap consistently captures a meaningful part/concept in the examples.
Revised: We can see that heatmap consistently captures a meaningful part/concept in the examples.

Original: Currently, our memory consumption increases quickly with c here since the constraint matrix in our LP formulation is of size
Revised: Currently, our memory consumption increases quickly with c here since the constraint matrix in our LP formulation is of size

Original: O ( c 2 ) × O ( c 2 ) .
Revised: O ( c 2 ) × O ( c 2 ) .

Original: This makes our method only work for small c on a GPU with 11 GB memory (here, we use c = 20 ).
Revised: This makes our method only work for small c on a GPU with 11 GB memory (here, we use c = 20 ).

Original: This scaling issue can be possibly solved by utilizing sparsity in the constraint matrix, but the sparse matrix operations are currently not well supported on mainstream deep learning platforms like PyTorch and Tensorﬂow.
Revised: This scaling issue can be possibly solved by utilizing sparsity in the constraint matrix, but the sparse matrix operations are currently not well supported on mainstream deep learning platforms like PyTorch and Tensorﬂow.

Original: Since our method provides backward gradients for the NMF operation, the heatmap generated here can, in fact, be used to construct a loss function during training in order to learn a interpretable models.
Revised: Since our method provides backward gradients for the NMF operation, the heatmap generated here can, in fact, be used to construct a loss function during training in order to learn a interpretable models.

Original: Objective
Revised: Objective

Original: [Equation]
Revised: [Equation]

Original: - p
Revised: - p

Original: [Equation]
Revised: [Equation]

Original: f
Revised: f

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: - p ij = ( f ‡ ij − (cid:15) ) - - -
Revised: - p ij = ( f ‡ ij − (cid:15) ) - - -

Original: F
Revised: F

Original: [Table]
Revised: [Table]

Original: A. V ERIFICATION OF U NROLLING GRADIENT AND THE ONE PROVIDED BY ˜ A − 1 ˜ b
Revised: A. V ERIFICATION OF U NROLLING GRADIENT AND THE ONE PROVIDED BY ˜ A − 1 ˜ b

Original: We use Fscore formulation as an example.
Revised: We use Fscore formulation as an example.

Original: For input sample x , the neural network predicts a score f ( x ) , and then the scores of a batch of samples will be used in solving the linear programming form of Fscore and be used to construct the loss function.
Revised: For input sample x , the neural network predicts a score f ( x ) , and then the scores of a batch of samples will be used in solving the linear programming form of Fscore and be used to construct the loss function.

Original: We compute the gradient from the ﬁnal loss function back to the predicted scores from the neural network and compare two approaches: one is that we use z = ˜
Revised: We compute the gradient from the ﬁnal loss function back to the predicted scores from the neural network and compare two approaches: one is that we use z = ˜

Original: A − 1 ˜ b as the solution (the one we used in our experiment) where we can compute gradient by only one step, another one is that we directly use y t resulting from the Newton iterations as the solution and compute gradients by unrolling those iterations.
Revised: A − 1 ˜ b as the solution (the one we used in our experiment) where we can compute gradient by only one step, another one is that we directly use y t resulting from the Newton iterations as the solution and compute gradients by unrolling those iterations.

Original: We then compute the cosine value between these two gradient vectors.
Revised: We then compute the cosine value between these two gradient vectors.

Original: By experiments on 100 randomly sampled batches, the average cosine value is 0 . 9991 , which means the two gradients are highly consistent.
Revised: By experiments on 100 randomly sampled batches, the average cosine value is 0 . 9991 , which means the two gradients are highly consistent.

Original: B P ROOFS AND D ETAILS OF R ESULTS IN S ECTION
Revised: B P ROOFS AND D ETAILS OF R ESULTS IN S ECTION

Original: In this section, we will provide the missing proofs and additional calculations in Section 3.
Revised: In this section, we will provide the missing proofs and additional calculations in Section 3.

Original: B. P ROOF OF L EMMA 1.
Revised: B. P ROOF OF L EMMA 1.

Original: Lemma 1 is restated here for convenience.
Revised: Lemma 1 is restated here for convenience.

Original: Lemma 3.
Revised: Lemma 3.

Original: Assume that A ∈ R m × n is a random matrix, and ﬁx some y ∈ R n .
Revised: Assume that A ∈ R m × n is a random matrix, and ﬁx some y ∈ R n .

Original: Then with probability one, g in equation 6 is quadratic (given by ˜ H, ∇ g ( y ) ) over a sufﬁciently small neighborhood of y .
Revised: Then with probability one, g in equation 6 is quadratic (given by ˜ H, ∇ g ( y ) ) over a sufﬁciently small neighborhood of y .

Original: Proof.
Revised: Proof.

Original: Using the integral form of second order Taylor’s expansion of σ 2 ( y )
Revised: Using the integral form of second order Taylor’s expansion of σ 2 ( y )

Original: = (max(0 , y )) 2 , we can show that,
Revised: = (max(0 , y )) 2 , we can show that,

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: See Remark 1 in Golikov & Kaporin (2019) for details.
Revised: See Remark 1 in Golikov & Kaporin (2019) for details.

Original: Without loss of generality, we can assume b = 0 by simply translating the origin.
Revised: Without loss of generality, we can assume b = 0 by simply translating the origin.

Original: Following the same remark, the diagonal matrix coincides with the step function based diagonal in ˜ H under the following condition on h :
Revised: Following the same remark, the diagonal matrix coincides with the step function based diagonal in ˜ H under the following condition on h :

Original: [Equation]
Revised: [Equation]

Original: Since y is ﬁxed, assuming that the entries of A are chosen from a continuous distribution such that e Tj A is uniformly distributed over the sphere, then ( e Tj Ah ) 2 follows a Beta (cid:0) 12 , n − 1 2 (cid:1) when h is drawn uniformly at random from the unit sphere, independent of A .
Revised: Since y is ﬁxed, assuming that the entries of A are chosen from a continuous distribution such that e Tj A is uniformly distributed over the sphere, then ( e Tj Ah ) 2 follows a Beta (cid:0) 12 , n − 1 2 (cid:1) when h is drawn uniformly at random from the unit sphere, independent of A .

Original: This means that no matter what y is, there exists sufﬁciently small h such that the left hand side of equation 20 is false with probability one, and in that neighborhood diag ( d ) =
Revised: This means that no matter what y is, there exists sufﬁciently small h such that the left hand side of equation 20 is false with probability one, and in that neighborhood diag ( d ) =

Original: ˜ H .
Revised: ˜ H .

Original: B. P ROOF OF T HEOREM
Revised: B. P ROOF OF T HEOREM

Original: Theorem 4.
Revised: Theorem 4.

Original: Assume that the primal LP has a unique optimal solution, and that the level set { x : Ax ≤ b, c T x ≤ α } is bounded for all α (for dual feasibility).
Revised: Assume that the primal LP has a unique optimal solution, and that the level set { x : Ax ≤ b, c T x ≤ α } is bounded for all α (for dual feasibility).

Original: Then short step (no line search)
Revised: Then short step (no line search)

Original: Newton’s method converges globally at a linear rate with local quadratic convergence.
Revised: Newton’s method converges globally at a linear rate with local quadratic convergence.

Original: Proof.
Revised: Proof.

Original: First, since the objective function is piecewise quadratic since it is a sum of piecewise quadratic functions.
Revised: First, since the objective function is piecewise quadratic since it is a sum of piecewise quadratic functions.

Original: In particular, it is self concordant since its third derivative zero everywhere.
Revised: In particular, it is self concordant since its third derivative zero everywhere.

Original: Now setting ρ < (cid:15) , we see that an approximate solution of the problem with the modiﬁed Hessian is also an approximate solution to equation 6.
Revised: Now setting ρ < (cid:15) , we see that an approximate solution of the problem with the modiﬁed Hessian is also an approximate solution to equation 6.

Original: Moreover, since the possible values of ˜ H is ﬁnite, the local norm (also known as Newton’s decrement) ∇ g ( y ) T ˜ H ( y ) − 1 ∇ g ( y ) is ﬁnite.
Revised: Moreover, since the possible values of ˜ H is ﬁnite, the local norm (also known as Newton’s decrement) ∇ g ( y ) T ˜ H ( y ) − 1 ∇ g ( y ) is ﬁnite.

Original: Hence, we can choose ρ >so that there is a descent direction d , that is, there exist a step size λ > 0 such that λ ∇ g ( x ) T d < 0 .
Revised: Hence, we can choose ρ >so that there is a descent direction d , that is, there exist a step size λ > 0 such that λ ∇ g ( x ) T d < 0 .

Original: Finally, we use Theorem 4.1.12 in Nesterov (2013) to claim the desired result.
Revised: Finally, we use Theorem 4.1.12 in Nesterov (2013) to claim the desired result.

Original: The assumptions in Theorem 4 are standard: 1.
Revised: The assumptions in Theorem 4 are standard: 1.

Original: uniqueness can easily be satisﬁed by randomly perturbing the cost vector; 2.
Revised: uniqueness can easily be satisﬁed by randomly perturbing the cost vector; 2.

Original: in most of our formulations, we explicitly have bound constraints on the decision variables, hence level sets are bounded.
Revised: in most of our formulations, we explicitly have bound constraints on the decision variables, hence level sets are bounded.

Original: B. D IFFERENTIATING THE S TEP F UNCTION IN R EMARK
Revised: B. D IFFERENTIATING THE S TEP F UNCTION IN R EMARK

Original: We will use a slightly modiﬁed “sufﬁx" notation as in Brookes (2005) in our calculations.
Revised: We will use a slightly modiﬁed “sufﬁx" notation as in Brookes (2005) in our calculations.

Original: That is, for a matrix A , (cid:126)A is the same as vec ( A ) , vectorization of A obtained by concatenating all the columns.
Revised: That is, for a matrix A , (cid:126)A is the same as vec ( A ) , vectorization of A obtained by concatenating all the columns.

Original: The following three properties relating the Kronecker product, (cid:126) · , and differentials will be used often:
Revised: The following three properties relating the Kronecker product, (cid:126) · , and differentials will be used often:

Original: Fact 1: For two vectors a, b , a ⊗ b = −−→ ba T . 2.
Revised: Fact 1: For two vectors a, b , a ⊗ b = −−→ ba T . 2.

Original: Fact 2: If A is p × q matrix, and B is a m × n matrix, then −→ ∂B = ( ∂B/∂A ) −→ ∂A where ∂B/∂A is the ( mn ) × ( pq ) Jacobian matrix of (cid:126)B with respect to (cid:126)A .
Revised: Fact 2: If A is p × q matrix, and B is a m × n matrix, then −→ ∂B = ( ∂B/∂A ) −→ ∂A where ∂B/∂A is the ( mn ) × ( pq ) Jacobian matrix of (cid:126)B with respect to (cid:126)A .

Original: If A or B is a column vector or scalar, then (cid:126) · has no effect.
Revised: If A or B is a column vector or scalar, then (cid:126) · has no effect.

Original: Fact 3: −−−−−−→ ∂ ( AXB ) =
Revised: Fact 3: −−−−−−→ ∂ ( AXB ) =

Original: (cid:0) B T ⊗ A (cid:1) −→ ∂X .
Revised: (cid:0) B T ⊗ A (cid:1) −→ ∂X .

Original: Using the above two facts, we can compute all the gradients needed to backpropagate through the unrolled iterations.
Revised: Using the above two facts, we can compute all the gradients needed to backpropagate through the unrolled iterations.

Original: We will show the computation for the gradient of Q − 1 u with respect to
Revised: We will show the computation for the gradient of Q − 1 u with respect to

Original: A ∈ R m × n for a ﬁxed u ∈ R n .
Revised: A ∈ R m × n for a ﬁxed u ∈ R n .

Original: We can apply chain rule to the following composition:
Revised: We can apply chain rule to the following composition:

Original: [Equation]
Revised: [Equation]

Original: to get, J f 2 ◦ f 1 = J f 2 ◦ J f 1 .
Revised: to get, J f 2 ◦ f 1 = J f 2 ◦ J f 1 .

Original: Now using Fact 2 on ∂ (cid:0) X − 1 (cid:1) =
Revised: Now using Fact 2 on ∂ (cid:0) X − 1 (cid:1) =

Original: − X − 1 ( ∂X ) X − 1 and some algebraic manipulation, we obtain,
Revised: − X − 1 ( ∂X ) X − 1 and some algebraic manipulation, we obtain,

Original: [Equation]
Revised: [Equation]

Original: We will now compute −→ J f 1 .
Revised: We will now compute −→ J f 1 .

Original: Note that ˜ H is also a function of A , so using product rule, we can write −→
Revised: Note that ˜ H is also a function of A , so using product rule, we can write −→

Original: J f 1 as a sum of three derivatives – with respect to each of A, A T , ˜ H .
Revised: J f 1 as a sum of three derivatives – with respect to each of A, A T , ˜ H .

Original: The derivatives with respect to A and A T are fairly straightforward to compute, so will focus on computing the derivative with respect to ˜ H .
Revised: The derivatives with respect to A and A T are fairly straightforward to compute, so will focus on computing the derivative with respect to ˜ H .

Original: To that end, we will use Fact 3, and show to compute the derivative of the step function by approximating it using the logistic function.
Revised: To that end, we will use Fact 3, and show to compute the derivative of the step function by approximating it using the logistic function.

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Note that these derivatives are used in computing derivatives of upstream network, so using distributional derivatives, and another application of chain rule to the left hand side of equation 22 results in the dirac delta function which is atomic, that is, has all its mass in a measure zero set.
Revised: Note that these derivatives are used in computing derivatives of upstream network, so using distributional derivatives, and another application of chain rule to the left hand side of equation 22 results in the dirac delta function which is atomic, that is, has all its mass in a measure zero set.

Original: Hence this calculation provides an mathematical justiﬁcation that the set of nondifferentiable points has measure zero for our training purposes.
Revised: Hence this calculation provides an mathematical justiﬁcation that the set of nondifferentiable points has measure zero for our training purposes.

Original: It is easy to formally verify this argument using differentiable tent functions as approximations to the heaviside step function.
Revised: It is easy to formally verify this argument using differentiable tent functions as approximations to the heaviside step function.