Wasserstein Logistic Regression with Mixed Features
Anonymous Author(s)
AffiliationAddress email
Abstract
Recent work has leveraged the popular distributionally robust optimizationparadigm to combat overfitting in classical logistic regression.
While the resultingclassification scheme displays a promising performance in numerical experiments,it is inherently limited to numerical features.
In this paper, we show that distributionally robust logistic regression with mixed ( i.e. , numerical and categorical)features, despite amounting to an optimization problem of exponential size, admits a polynomial-time solution scheme.
We subsequently develop a practicallyefficient column-and-constraint approach that solves the problem as a sequence ofpolynomial-time solvable exponential conic programs.
Our model retains manyof the desirable theoretical features of previous works, but—in contrast to theliterature—it does not admit an equivalent representation as a regularized logisticregression, that is, it represents a genuinely novel variant of logistic regression.
We show that our method outperforms both the unregularized and the regularizedlogistic regression on categorical as well as mixed-feature benchmark instances.
Introduction
Consider a data set ( x i , y i )
Ni =1 with feature vectors x i and associated binary labels y i ∈ {− 1 , 1 } .
Classical logistic regression assumes that the labels depend probabilistically on the features via
Prob ( y | x ) = (cid:2) 1 + exp( − y · [ β 0 + β ⊤ x ]) (cid:3) − 1 , where the parameters ( β 0 , β ) ∈ R 1+ n are estimated from the empirical risk minimization problemminimize ( β 0 , β )
N
N (cid:88) i
= l β ( x i , y i ) subject to ( β 0 , β ) ∈ R n + with the log-loss function l β ( x , y ) :
= log (cid:0) 1 + exp (cid:0) − y · [ β 0 + β ⊤ x ] (cid:1)(cid:1) .
Its compelling performanceacross many domains, the availability of mature and computationally efficient algorithms as well asits interpretability have all contributed to the widespread adoption of logistic regression [6, 18, 27].
Similar to other machine learning models, logistic regression is prone to overfitting, especially whenthe number of training samples is small relative to the number of considered features.
Moreover,logistic regression can be sensitive to erroneous feature and label values as well as distribution shiftsunder which the training and test sets stem from different distributions.
In recent years, distributionallyrobust (DR) optimization [3, 4] has been proposed to simultaneously address these challenges.
Tothis end, the DR optimization paradigm models a machine learning task as a zero-sum game betweenthe decision maker, who seeks to obtain the most accurate model ( e.g. , the parameter vector βin a logistic regression), and a fictitious adversary who observes the decision maker’s model and subsequently selects the worst data-generating distribution in the vicinity of the empirical distribution formed from the available training data.
In this context, the similarity of distributions is commonlymeasured in terms of moment bounds [9, 39], ϕ -divergences such as the Kullback-Leibler divergence[19, 22] or the Wasserstein distance [15, 25].
It has been observed that in many cases, the resulting DR machine learning models are equivalent to regularized versions of the same models, in whichcase distributionally robustness provides a new perspective on regularizers that are often selected adhoc [21, 41].
Most importantly, DR machine learning models often admit computationally efficientreformulations as finite-dimensional convex optimization problems that can be solved in polynomialtime with off-the-shelf solvers.
In this work we study a DR variant of the logistic regression where the distance between theempirical distribution and the unknown true data-generating distribution is measured by the popular Wasserstein (also known as Kantorovich-Rubinstein or earth mover’s) distance [15, 25].
This problemhas been first studied by [31], who show that the resulting DR logistic regression problem admitsan equivalent reformulation as a polynomial-size convex optimization problem if all features arenumerical .
Applying similar techniques to the mixed-feature logistic regression problem, whichappears to be most prevalent in practice, would result in an exponential-size convex optimizationproblem whose naïve solution with the methods of [31] does not scale to interesting problem sizes( cf.
Section 4).
Our contributions may be summarized as follows.
(i)
On the theoretical side , we show that the complexity of DR mixed-feature regression
crucially relies on the selected loss function.
In particular, for the log-loss function employed
in logistic regression, the problem—despite its natural representation as an exponential-size
convex optimization problem—admits a polynomial-time solution scheme.
We also show that in stark contrast to earlier variants of the problem, our mixed-feature regression does
not admit an equivalent representation as a regularized problem. This provides compelling evidence that the DR logistic regression problem with mixed features is fundamentally different to the DR problem with only numerical features.
(ii)
On the computational side , we propose a column-and-constraint scheme that solves the DRmixed
-feature logistic regression as a sequence of
polynomial -time solvable exponential conic programs.
We show that the key step of our procedure, the identification of the most
violated constraint , can
be implemented
efficiently for a broad range of metrics, despite itsnatural
representation
as a combinatorial optimization problem.
Indeed, identifying the mostviolated constraint by brute force is out of the question
, while the standard approach [43] ofsolving the most violated constraint problem would have an unacceptably
high
runtime (iii) On the numerical side
we show that our column-and-constraint scheme drastically reduces
computation times over a naïve monolithic implementation of the regression problem
We
also show that our model performs favorably on standard categorical and mixed-feature
benchmark instances when compared against classical and regularized logistic regression
The literature on DR machine learning under the Wasserstein distance is vast and rapidly growing.
Recent works have considered, among others, the use of Wasserstein DR models in multi-labellearning [14], generative adversarial networks [2, 8] and the generation of adversarial examples [40],density estimation [38] and learning Gaussian mixture models [20], graph-based semi-supervisedlearning [35], supervised dimensionality reduction [13] and reinforcement learning [1]. DR has alsobeen found to help alleviate problems with overfitting [24, 21], label uncertainty [21] and distributionshifts [37, 36].
We refer to [21] for a recent review of the literature.
Our work is most closely relatedto [31], and we compare our findings with the results of that work in Section 2.
We proceed as follows.
Section 2 defines and analyzes the mixed-feature DR logistic regressionproblem, which is solved in Section 3 via column-and-constraint generation.
We report numericalresults in Section 4.
Auxiliary material and all proofs are relegated to the appendix.
Notation.
We define B = { 0 , 1 } and [ N ] = { 1 , . . .
, N } for N ∈ N .
The set of all probabilitydistributions supported on a set Ξ is denoted by P 0 (Ξ) , while the Dirac distribution placing unitprobability mass on x ∈ R n is denoted by δ x ∈ P 0 ( R n ) .
The indicator function 1 [ E ] attains the value1 ( 0 ) whenever the expression E is (not) satisfied.
Finally, we use Roman d to denote differentialsand to distinguish them from the d used to denote distances.
Mixed-Feature DR Logistic Regression
Section 2.1 derives an exponential-size convex programming formulation of the DR logistic regressionproblem that serves as the basis of our analysis.
Section 2.2 compares our formulation with a naïvemodel that treats categorical features as continuous ones.
Section 2.3, finally, shows that ourformulation, despite its exponential size, can be solved in polynomial time due to the benign structureof the log-loss function.
It also establishes that in stark contrast to the literature, our formulation doesnot reduce to a regularized non-robust logistic regression, that is, it constitutes a genuinely novelvariant of the logistic regression problem.
Our formulation enjoys strong finite-sample and asymptotic performance guarantees from the literature.
Moreover, despite its exponential size, our model always accommodates worst-case distributionsthat exhibit a desirable sparsity pattern.
We relegate these results to Appendix A.
2.1 Formulation as an Exponential-Size Convex Program
From now on, we consider a mixed-feature data set { ξ i :=
( x i , z
i , y i ) } i ∈ [ N ] with n numericalfeatures x i =
( x i 1 , . . .
, x in ) ∈ R n , m categorical features z
i =
( z i 1 , . . .
, z im ) ∈ C ( k 1 ) × . . .
× C ( k m )and a binary label y i ∈ {− 1 , +1 } .
Here, C ( s ) = { z ∈ B s − 1 : (cid:80) j ∈
[ s − 1]
z j ≤ 1 } for s ∈ N \ { 1 }represents the one-hot encoding of a categorical feature with s possible values; in particular, C (2) =
Bencodes a binary feature.
We denote by C =
C ( k 1 ) × . . .
× C ( k m ) and Ξ = R n × C × {− 1 , +1 } thesupport of the categorical features as well as the data set, respectively, and we let k = k 1 + . . .
+ k m − mbe the number of slopes used for the all categorical features.
If we had access to the true data-generating distribution P 0 ∈ P 0 (Ξ) , we would solve the (non-robust)logistic regression problemminimize
[Equation]
subject to
[Equation]
where E P 0 denotes the expectation under P 0 , and where the log-loss function l β now takes the form
[Equation]
to account for the presence of categorical features.
Since the distribution P 0 is unknown in practice, theempirical risk minimization problem replaces P 0 with the empirical distribution (cid:98) P N :
= 1 N (cid:80) Ni =
δ ξ ithat places equal probability mass on all observations { ξ i } i ∈ [ N ] .
Standard arguments show thatwhen these observations are i.i.d., the empirical risk minimization problem recovers the logisticregression under P 0 as N −→ ∞ .
In practice, however, data tends to be scarce, and the empiricalrisk minimization problem exhibits an ‘optimism bias’ that is also known as overfitting [6, 18, 27],the error maximization effect of optimization [10, 23] or the optimizer’s curse [34].
DR logistic regression combats the aforementioned overfitting phenomenon by solving the semiinfinite optimization problem
[Equation]
where the ambiguity set B ϵ ( (cid:98) P N ) contains all distributions Q in a (soon to be defined) vicinity of theempirical distribution (cid:98) P N , and where the expectation is taken with respect to Q .
Problem (1) can beinterpreted as a zero-sum game between the decision maker, who chooses a logistic regression modelparameterized by β , and a fictitious adversary that observes β and subsequently chooses the ‘worst’distribution (in terms of the incurred log-loss) from B ϵ ( (cid:98) P N ) .
Contrary to the classical non-robustlogistic regression, problem (1) guarantees to over estimate the log-loss incurred by β under theunknown true distribution P
as long as P 0 is contained in B ϵ ( (cid:98) P N ) .
On the other hand, problem (1)recovers the empirical risk minimization problem when the ambiguity set B ϵ ( (cid:98) P N ) approaches asingleton set that only contains the empirical distribution (cid:98) P N .
Note that problem (1) is convex asthe convexity of the logistic regression objective E Q [ l β ( x , z , y )] is preserved under the supremumoperator.
That said, the problem typically constitutes a semi-infinite program as it comprises finitely many decision variables
but—if the embedded supremum in the objective is brought to the constraintsvia an epigraph reformulation—infinitely many constraints whenever the ambiguity set harborsinfinitely many distributions.
As such, it is not obvious how the problem can be solved efficiently.
In this paper, we choose as ambiguity set the Wasserstein ball B ϵ ( (cid:98) P N ) :
= { Q ∈ P 0 (Ξ) :
W( Q , (cid:98) P N ) ≤ ϵ } of radius ϵ > 0 that is centered at the empirical distribution (cid:98) P N .
Definition 1 (Wasserstein Distance) .
The type-1 Wasserstein ( Kantorovich-Rubinstein , or earthmover’s ) distance between two distributions P ∈ P 0 (Ξ) and Q ∈ P
(Ξ) is defined as
[Equation]
where ξ = ( x , z , y ) ∈ Ξ and ξ ′
=
( x ′ , z ′ , y ′ )
∈ Ξ , while d ( ξ , ξ ′ ) is the ground metric on Ξ .
The Wasserstein distance can be interpreted as the minimum cost of moving Q to P when d ( ξ , ξ ′ )
isthe cost of moving a unit mass from ξ to ξ ′ .
The Wasserstein radius ϵ thus imposes a budget on thetransportation cost that the adversary can spend on perturbing the empirical distribution (cid:98) P N .
We next define the ground metric d that we use throughout the paper.
Definition 2 (Ground Metric) .
We measure the distance between two data-points ξ =
( x , z , y ) ∈ Ξand ξ ′
=
( x ′ , z ′ , y ′ )
∈ Ξ with z = ( z 1 , . . .
, z m ) and z ′ =
( z ′ 1 , . . .
, z ′ m ) as
[Equation]
where ∥·∥ is any rational norm on R n , κ > 0 and the metric d C on C satisfies
[Equation]
for some p >
[Equation]
3b
[Equation]
Intuitively, the ground metric d measures the distance of the numerical features x and x ′ by the normdistance ∥ x − x ′ ∥ , whereas the distance of the categorical features z and z ′ is measured by (a powerof) the number of discrepancies between z and z ′ .
Likewise, any discrepancy between the labels yand y ′ is accounted for by a constant κ , which allows for scaling between the features and the labels.
We first extend the convex optimization model of [31] for the DR continuous-feature logistic regression to an exponential conic model that accommodates for categorical features.
Theorem 1 (Exponential Conic Representation) .
The DR logistic regression problem (1) admits theequivalent reformulationminimize β ,λ, s , u ± , v ± λϵ + 1 N (cid:88) i ∈ [ N ] s i subject to u + i, z + v
+ i, z ≤( u + i, z , 1 , − s i − λd C
( z , z i ))
∈ K exp ( v + i, z , 1 , − y i β N ⊤ x i − y i β C ⊤ z − y i β 0 − s i − λd C ( z , z i )) ∈ K exp  
∀
[Equation]
∈ [ N ] , ∀
[Equation]
∈ C u − i, z + v − i, z ≤( u − i, z , 1 , − s i − λd C ( z , z i ) − λκ ) ∈ K exp ( v − i, z , 1 , y i β N ⊤ x i + y
i β C ⊤
z
+ y i β 0 − s i
[Equation]
and z [Equation] as a finite-dimensional exponential conic program, where K exp denotes the exponential cone K
[Equation]
The s i ’s and λ that appear in (4) are dual variables that arise from dual of the inner ( i.e. , sup ) problemin (1).
For each z ∈ C and i ∈ [ N ] , ( u + i, z , v
+ i, z , u − i, z , v − i, z ) is a vector of auxiliary variables that we useto model softplus constraints that arise from our intermediate analysis of problem (1) (Appendix E).
Problem (4) is finite-dimensional and convex.
Moreover, since exponential conic programs admit aself-concordant barrier function, they can be solved in polynomial time relative to their input size[28].
Due to the presence of categorical features in our regression problem, however, problem (4)comprises exponentially many variables and constraints, and a naïve solution of (4) would thereforerequire an exponential amount of time.
The exponential size of problem (4) renders our formulationfundamentally different from the model of [31], and it significantly complicates the solution.
2.2 Comparison with Continuous-Feature DR Logistic Regression
Our formulation (4) of the mixed-feature DR logistic regression (1) accounts for categorical featuresz ∈ C at the expense of an exponential number of variables and constraints.
It is therefore temptingto treat the categorical features as continuous ones and directly apply the continuous-feature-onlyreformulation of the DR logistic regression (1) proposed by [31].
In the following, we argue thatsuch a reformulation would hedge against nonsensical worst-case distributions that in turn lead tooverly conservative regression models.
To see this, consider a stylized setting where the data set( z i , y i )
i ∈
[ N ] comprises a single binary feature z i ∈ {− 1 , +1 } that impacts the label y i ∈ {− 1 , +1 }via the logistic model
[Equation]
with β = 1 .
(While we use z i ∈ { 0 , 1 } in the other sections of the paper, the analysis and results ofthis subsection and Appendix B assumed z i ∈ {− 1 , +1 } as this made comparisons with [31] easier.)
We attempt to recover this logistic model in two ways:
Mixed-feature model.
We employ our DR logistic regression (1) with a single categorical feature, using p = 1 and κ = 1 in the ground metric ( cf.
Definition 2).
Continuous-feature model.
We employ a variant of problem (1) that treats the categoricalfeature
z i as a continuous one.
We choose as ground metric
[Equation]
We therefore use the feature distances
[ z ̸ = z ′ ] in the mixed-feature case and 1 2 | z − z ′ | in thecontinuous-feature case.
Note they both attain the value 0 ( 1 ) for equal (different) feature values.
Figure 1 (left) compares the mean values (dashed lines) as well as the 15% and 85% quantiles (shadedregions) of β for the standard logistic regression, our mixed-feature as well as the continuous-featuremodel as a function of the size N of the data set.
For both DR models, we employed the same Wasserstein radius ϵ ∝ 1 / √ N , which is motivated by the finite sample guarantee presented in thenext section ( cf.
Theorem 6).
The figure shows that the continuous-feature model excessively shrinksits estimates of β , thus leading to overly conservative results.
In fact, the true parameter value β = 1 consistently lies outside the confidence region of the continuous-feature model, independent of thesample size N .
This is due to the fact that the continuous-feature model accounts for nonsensicalworst-case distributions under which the categorical feature can take values outside its domain{− 1 , +1 } , see Figure 1 (right).
In fact, the continuous-feature model places non-zero probabilityon an extreme scenario under which the binary feature z ∈ {− 1 , +1 } attains the value 15 , 312 . (In Appendix B we explain why this value of 15 , 312 arises.)
Our mixed-feature model, on the otherhand, restricts the worst-case distribution to the domain of the categorical feature and thus hedgesagainst realistic distributions only.
We note that in practice, the radius ϵ of the Wasserstein ball will be chosen via cross-validation( cf.
Section 4), in which case our mixed-feature model reliably outperforms the classical logisticregression on standard benchmark instances.
While one may argue that the impact of nonsensicalworst-case distributions in the continuous-feature model is alleviated by the cross-validated radii,
Figure 1 offers strong theoretical and practical reasons against the use of a continuous-feature modelfor categorical or mixed-feature problems: The continuous-feature model hedges against a clumsilyperturbed worst-case distribution that introduces a pronounced bias in the estimation.
We also notethat another possible approach would be to use the continuous-feature model but to restrict the supportof binary features to [ − 1 , 1] . Unfortunately, for the log-loss function no tractable reformulationwith support constraints appears to be known; see [32] .
Moreover, Section 4 will show that themixed-feature model, despite its exponential size, can be solved quickly and reliably with our novelcolumn-and-constraint approach from Section 3.
2.3 Complexity Analysis
We first show that despite its exponential size, the DR logistic regression problem (1) admits apolynomial-time solution.
Key to this perhaps surprising finding is the shape of the loss function:while problem (1) is strongly NP-hard (and thus unlikely to admit a polynomial-time solution scheme)for generic loss functions, it can be solved in polynomial time for the log-loss function l β employedin logistic regression.
Theorem 2 (Complexity of the DR Logistic Regression (1)) .
(i) For generic loss functions l β , problem (1) is strongly NP-hard even if n = 0 and N = 1 .
(ii)
For the loss function l β ( x , z , y ) =
log (cid:0) 1 + exp (cid:2) − y · (cid:0) β 0 + β N ⊤ x + β C ⊤ z
(cid:1)(cid:3)(cid:1) and theground metric of Definition 2, problem (1) can be solved to δ -accuracy in polynomial time
Recall that an optimization problem is solved to δ -accuracy if a δ -suboptimal solution is identified thatsatisfies all constraints modulo a violation of at most δ .
The consideration of δ -accurate solutions isstandard in the numerical solution of nonlinear programs where an optimal solution may be irrational.
A by now well-known result shows that when m =
(no categorical features), the DR logisticregression problem (1) reduces to a classical logistic regression with an additional regularizationterm ∥ β x ∥ ∗ in the objective function when the output label weight κ in Definition 2 approaches ∞[31, 32].
We next show that this reduction to a classical regularized logistic regression no longerholds in our problem setting when categorical features are present.
Theorem 3 (Absence of a Reformulation as a Regularized Problem) .
Even when the output labelweight κ approaches ∞ in the DR logistic regression (1) , problem (1) does not admit an equivalentreformulationminimize
[Equation]
subject
[Equation] as a classical regularized logistic regression for any regularizer R : R 1+ n + k
→ R .
Note that Theorem 3 does not only preclude the existence of a specific regularizer, but it excludes theexistence of any regularizer, no matter how complex its
Algorithm 1 Column-and-Constraint Generation Scheme for Problem (4).
Set LB 0 = −∞ and UB 0 = + ∞
Choose (possibly empty) subsets W + , W − ⊆
[ N ] × C while LB t ̸ = UB t do
Let θ ⋆ be the optimal value and ( β , λ, s , u ± , v ± ) be a minimizer of problem (4) where:
• the first constraint set is only enforced for all ( i, z ) ∈ W + ; • the second constraint set is only enforced for all ( i, z ) ∈ W − ; • we only include those ( u + i, z , v
+ i, z ) for which ( i, z ) ∈ W + ; • we only include those ( u − i, z , v − i, z ) for which ( i, z ) ∈ W − .
Identify a most violated index ( i + , z + ) of the first constraint set in (4) and add it to W +
Identify a most violated index ( i − , z − ) of the second constraint set in (4) and add it to W −
Let ϑ + and ϑ − be the violations of ( i + , z + ) and ( i − , z − ) , respectively
Update LB t = θ ⋆ and UB t = min { UB t − 1 , θ ⋆ + log(1 + max { ϑ + , ϑ − } ) }
Update t = t +end while
Algorithm 2 Identification of Most Violated Constraints in the Reduced Problem (4).
for j ∈ { 1 , . . .
, m } do
Find a feature value z ⋆j that minimizes y i · β C ,j ⊤ z j across all z j ∈ C ( k j ) \ { z ij } end for
Let π : [ m ] → [ m ] be an ordering such that y iπ ( j ) · β C ,π ( j ) ⊤ ( z [Equation]
( j ) − z iπ ( j ) ) ≤ y iπ ( j ′ ) · β C ,π ( j ′ )
⊤ ( z ⋆π ( j ′ )
− z iπ ( j ′ ) ) ∀ 1 ≤ j ≤ j ′ ≤ m
Set W = ∅ for δ ∈ { 0 , 1 , . . .
, m } do
Update W = W ∪ { z
where z j = z ⋆j if π ( j ) ≤ δ and z j = z ij otherwise end for Determine the most violated constraint from the candidate set W  Column-and-Constraint Solution Scheme Our DR logistic regression problem (4) comprises exponentially many variables and constraints,which renders its solution as a monolithic exponential conic program challenging.
Instead, Algorithm 1 employs a column-and-constraint generation scheme, e.g. , [43], that alternates between (i) thesolution of relaxations of problem (4) that omit most of its variables and constraints and (ii) addingthose variables and constraints that promise to maximally tighten the relaxations. Theorem 4.
Algorithm 1 solves problem (4) in finitely many iterations. Moreover, LB t and UB tconstitute monotone sequences of lower and upper bounds on the optimal value of problem (4) . A key step in Algorithm 1 is the identification of most violated indices ( i, z ) ∈ [ N ] × C of the first andsecond constraint set in the reduced DR regression problem (4) for a fixed solution ( β , λ, s , u ± , v ± ) .
For the first constraint set in (4), the identification of such indices requires for each data point i ∈
[ N ]the solution of the combinatorial problemmaximize
[Equation]
(cid:21)(cid:27) subject to
[Equation]
(5) where an optimal value greater than 1 corresponds to a violated constraint; an analogous problem canbe defined for the second constraint set in (4).
Despite its combinatorial nature, the above problemcan be solved efficiently by means of Algorithm 2.
Theorem 5.
Algorithm 2 identifies a most violated constraint for a given data-point i ∈ [ N ] in time
O ( k + n + m 2 ) .
Algorithm 2 first determines for each categorical feature z j , j = 1 , . . .
, m , the feature value z ⋆j ∈
C ( k j ) \ { z ij } that, ceteris paribus , contributes to a maximal constraint violation in time O ( k ) .
Thesubsequent step sorts the different features j = 1 , . . .
, m in descending order of their contribution toconstraint violations in time O
( m log m ) .
The last step of Algorithm 2, finally, constructs the mostviolated constraint z across those in which δ = 0 , . . .
, m categorical feature values deviate from z iand subsequently picks the most violated constraint of these in time O ( n + m 2 ) .
We note that theoverall complexity of O ( k + n + m 2 ) can be reduced to O ( k + n + m log m ) by a clever use of datastructures in the final step; for ease of exposition, we omit the details.
Finally, since Algorithm 2 isapplied to each data point i ∈
[ N ] , the overall complexity increases by a factor of N .
We provideadditional intuition behind Algorithm 2 immediately before the proof of Theorem 5 in Appendix E.
Numerical Results
Section 4.1 first compares the runtimes of our column-and-constraint scheme from Section 3 withthose of solving the DR logistic regression problem (4) naïvely as a monolithic exponential conicprogram.
We subsequently compare the classification performance of problem (1) with those of aclassical unregularized and regularized logistic regression on standard benchmark instances withcategorical features (Section 4.2) and mixed features (Appendix C).
All algorithms were implemented in Julia [5] (MIT license) and executed on Intel Xeon 2.66GHzprocessors with 8GB memory in single-core mode.
We use MOSEK 9.3 [26] (commercial) to solveall exponential conic programs through JuMP [12] (MPL2 License). (We note the open source solvers
Ipopt and CVXOpt could be used instead of MOSEK.)
All source codes and detailed results areavailable on GitHub (URL withheld to maintain anonymity during the review process).
4.1 Runtime Comparison with Monolithic Formulation
We first compare the computation times of our column-and-constraint scheme from Section 3 withthose of solving the DR logistic regression problem (4) naïvely as a monolithic exponential conicprogram.
To this end, we randomly generate synthetic logistic regression instances with varyingnumbers N of data points and m of binary features.
While Figure 2 (left) shows that both approachesscale similarly in the number N of data points, Figure 2 (middle) reveals that the solution of themonolithic formulation scales exponentially in the number m of binary features.
In contrast, ourcolumn-and-constraint scheme scales gracefully in both the number N of data points and the numberm of binary features ( cf.
Figure 2, right).
A similar behavior can be observed with more general,non-binary categorical features; we omit the results due to space constraints.
4.2 Performance on Categorical-Feature Instances
We next compare the classification performance (in terms of the out-of-sample classification error) ofour unregularized (‘DRO’) and Lasso-regularized (‘r-DRO’) DR logistic regression problem (1) withthose of a classical unregularized (‘LR’) as well as Lasso-regularized (‘r-LR’), mass transportationregularized (MT) [31, 32], and robust Wasserstein profile inference-regularized (PI) [7] logistic [Table] regression on the 14 most popular UCI data sets that only contain categorical features having morethan 30 rows [11] (varying licenses).
Instances with multiple output labels are indicated with a star;we convert them into instances with binary output labels by distinguishing between the majority classvs. all other classes.
The instances vary in the number N of data points, the number m of categoricalfeatures as well as, accordingly, the number k of slopes considered in the one-hot encoding of thecategorical features ( cf.
Section 2.1).
All results are reported as means over 100 random trainingset-test set splits (80%:20%).
The radius ϵ ∈ { 0 , 10 − 5 , . . .
, 10 − 4 , . .
, 1 } of the Wasserstein ballas well as the Lasso penalty γ ∈ { 0 , 12 · 10 − 5 , . .
, 12 · 10 − 4 , . .
. , 12 } are selected via 5-fold crossvalidation.
We consider two variants of our DR logistic regression that employ a different output labelweight ( κ = 1 vs. κ = m ) in the ground metric ( cf.
Definition 2).
The results are reported in Table 1.(Run-times and are provided in Appendix D where we also describe how statistical significance wasassessed.)
The table shows that for the unregularized model, the classical logistic regression achievesthe lowest classification error in 21% of the instances, whereas our DR logistic regression achievethe lowest classification error in 36% and 79% of the instances for κ = 1 , m , respectively. (Thedouble-counting of ties means these percentages don’t sum to 100%.)
For the regularized model,the results change to 14% (classical logistic regression) vs. 57% (each of our models).
Within thenumerical-feature Wasserstein DRO benchmarks, PI achieves the lowest classification error in 14%of the instances, whereas MT achieves in 42% and 57% of the instances for κ = 1 , m , respectively.
Globally, one of our methods is the winning approach in 12 out of 14 datasets (in the remaining 2they are the second best approach without statistically significant inferiority).
In 9 of the datasetsour methods win strictly (without a tie), and in 7 of these datasets the improvements are statisticallysignificant over all other approaches.
Conclusions
We proposed a new DR mixed-feature logistic regression model where the proximity betweenthe empirical distribution and the unknown true data-generating distribution is measured by thepopular Wasserstein distance.
Despite its exponential-size formulation, we prove that the underlyingoptimization problem can be solved in polynomial time, and we develop a practically efficientcolumn-and-constraint generation scheme for its solution.
The promising performance of our modelis demonstrated in numerical experiments on standard benchmark instances.
We note that our columnand-constraint scheme readily extends to other DR mixed-feature machine learning models suchas linear regression, support vector machines and decision trees.
Applying our algorithm in thosesettings is a promising direction for future research.
A further interesting direction is the developmentof tailored solution schemes rather than using off-the-shelf solvers for solving our DR mixed-featurelogistic regression problem.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes] The abstract explains (i) the motivation for using
Wasserstein DR optimization in machine learning, (ii) recent successes of this approachin logistic regression with numerical features and (iii) the lack of work on the mixedfeature setting; these explanations are further elaborated upon in Section 1.
This sectionalso provides a high-level summary of our contributions, which are in line with thedevelopments in the remainder of the paper.(b)
Did you describe the limitations of your work?
[Yes] Our numerical results have beenconducted in an objective and reproducible way (choosing the most popular UCI datasets), and our experiments include data sets where the standard logistic regressionoutperforms our approach.(c)
Did you discuss any potential negative societal impacts of your work?
[N/A]
Our workextends a well-known machine learning method (DR logistic regression) to a new butwell-known setting (presence of categorical features).
In light of the ethics reviewguidelines, we therefore believe that our work does not have any direct implications onsociety beyond the ones that are already known.(d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes] We have carefully reviewed the ethics guideline on the NeurIPS 2022conference webpage.
We believe that our paper conforms to these guidelines.2.
If
you are
including
theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] All of our
lemmas
and theorems state the full set of assumptions, with exception
of the blanket
assumptions that are assumed to hold throughout the paper and that are clearly marked
as such.
(b) Did you include complete proofs of all theoretical results?
[Yes] All proofs are included
in the appendix of the paper.
If you ran experiments...
(a) Did you include the code, data, and instructions needed
to
reproduce the main experi-
mental results (either in the supplemental material or as a URL
[Yes
] All code, data
and instructions for our experimental results are published on GitHub.
To maintain
anonymity during the review process, we do not provide a link in the current version of
the paper.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] The most important choices are described in the numerical section,
while some of the details are relegated to the appendix.
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] The runtime results provide confidence regions across
multiple random runs, and we report error bars for all other experiments in the appendix.
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] Information about the employed
CPU is provided in the numerical results section; details about the high-performance
cluster that we used are relegated to the appendix.
Note that all experiments are run insingle-core mode.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We are citing UCI for
the data sets, MOSEK for the solution of the optimization problems as well as Julia(and JuMP) for the implementation of the algorithms.
(b) Did you mention the license of the assets?
[Yes] We mention the licenses of UCI,
MOSEK and Julia in the numerical results section.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
All code, data and instructions for our experimental results are published on GitHub.
To
maintain anonymity during the review process, we do not provide a link in the current
version of the
paper.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
[Table]
using/curating?
[N/A] We are using the most popular UCI machine learning data sets,
[Table]
for which to our best knowledge consent has already been given.
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
[Table]
information or offensive content?
[N/A] We are using the most popular UCI machine
[Table]
learning data sets, for which to our best knowledge this issue has already been resolved.
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
Our research does not have any crowdsourcing or human subject
[Table]
components.
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review Board
[Table]
(IRB) approvals, if applicable?
[N/A]
Our research does not have any crowdsourcing
[Table]
or human subject components.
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
Our research does not have any crowdsourc-
[Table]
ing or human subject components.
[Table]