Wasserstein Logistic Regression with Mixed Features
Anonymous Author(s)
AffiliationAddress email
Abstract
Recent work has leveraged the popular distributionally robust optimizationparadigm to combat overfitting in classical logistic regression.
While the resultingclassification scheme displays a promising performance in numerical experiments,it is inherently limited to numerical features.
In this paper, we show that distributionally robust logistic regression with mixed ( i.e. , numerical and categorical)features, despite amounting to an optimization problem of exponential size, admits a polynomial-time solution scheme.
We subsequently develop a practicallyefficient column-and-constraint approach that solves the problem as a sequence ofpolynomial-time solvable exponential conic programs.
Our model retains manyof the desirable theoretical features of previous works, but—in contrast to theliterature—it does not admit an equivalent representation as a regularized logisticregression, that is, it represents a genuinely novel variant of logistic regression.
We show that our method outperforms both the unregularized and the regularizedlogistic regression on categorical as well as mixed-feature benchmark instances.
Introduction
Consider a data set ( x i , y i )
Ni =1 with feature vectors x i and associated binary labels y i ∈ B := {− 1 , 1 } .
Classical logistic regression assumes that the labels depend probabilistically on the features via
Prob ( y | x ) = (cid:2) 1 + exp( − y · [ β 0 + β ⊤ x ]) (cid:3) − 1 , where the parameters ( β 0 , β ) ∈ R n +1 are estimated from the empirical risk minimization problemminimize ( β 0 , β )
N
N (cid:88) i
= l β ( x i , y i ) subject to ( β 0 , β ) ∈ R n + with the log-loss function l β ( x , y ) :
= log (cid:0) 1 + exp (cid:0) − y · [ β 0 + β ⊤ x ] (cid:1)(cid:1) .
Its compelling performanceacross many domains, the availability of mature and computationally efficient algorithms as well asits interpretability have all contributed to the widespread adoption of logistic regression [6, 17, 25].
Similar to other machine learning models, logistic regression is prone to overfitting, especially whenthe number of training samples is small relative to the number of considered features.
Moreover,logistic regression can be sensitive to erroneous feature and label values as well as distribution shiftsunder which the training and test sets stem from different distributions.
In recent years, distributionallyrobust optimization [3, 4] has been proposed to simultaneously address these challenges.
To thisend, the distributionally robust optimization paradigm models a machine learning task as a zero-sumgame between the decision maker, who seeks to obtain the most accurate model ( e.g. , the parametervector β in a logistic regression), and a fictitious adversary who observes the decision maker’smodel and subsequently selects the worst data-generating distribution in the vicinity of the empirical distribution formed from the available training data.
In this context, the similarity of distributionsis commonly measured in terms of moment bounds [8, 34], ϕ -divergences such as the KullbackLeibler divergence [18, 21] or the Wasserstein distance [14, 23].
It has been observed that in manycases, the resulting distributionally robust machine learning models are equivalent to regularizedversions of the same models, in which case distributionally robustness provides a new perspective onregularizers that are often selected ad hoc [20, 36].
Most importantly, distributionally robust machinelearning models often admit computationally efficient reformulations as finite-dimensional convexoptimization problems that can be solved in polynomial time with off-the-shelf solvers.
In this work we study a distributionally robust variant of the logistic regression where the distancebetween the empirical distribution and the unknown true data-generating distribution is measured bythe popular Wasserstein (also known as Kantorovich-Rubinstein or earth mover’s) distance [14, 23].
This problem has been first studied by [28], who show that the resulting distributionally robust logisticregression problem admits an equivalent reformulation as a polynomial-size convex optimizationproblem if all features are numerical .
Applying similar techniques to the mixed-feature logisticregression problem, which appears to be most prevalent in practice, would result in an exponentialsize convex optimization problem whose naïve solution with the methods of [28] does not scale tointeresting problem sizes ( cf.
Section 4).
Our contributions may be summarized as follows
(i)
On the theoretical side , we show that the complexity of distributionally robust mixed-feature
regression crucially relies on the selected loss function.
In particular, for the log-loss
function employed in logistic regression, the problem—despite its natural representation
as an exponential-size convex optimization problem—admits a polynomial-time solution scheme.
We also show that in stark contrast to earlier variants of the problem, our mixed-
feature regression does not admit an equivalent representation as a regularized problem.
(ii)
On the computational side , we propose a column-and-constraint scheme that solves the
distributionally robust mixed-feature logistic regression as a sequence of polynomial-
timesolvable exponential conic programs.
We show that the key step of our procedure, the identification
of the most violated constraint, can
be implemented efficiently
for a broadrange of metrics, despite its natural
representation as a
combinatorial optimization problem.
iii)
On the numerical side ,
we
show that our column and -constraint scheme drastically reduces
computation times over a naïve monolithic implementation of the regression problem.
We
also show that our model performs favorably on standard categorical and mixed-feature benchmark
instances when compared against classical and regularized logistic
regression. The literature on distributionally robust machine learning under the Wasserstein distance is vast andrapidly growing.
Recent works have considered, among others, the use of Wasserstein distributionallyrobust models in multi-label learning [13], generative adversarial networks [2, 7] and the generation ofadversarial examples [35], density estimation [33] and learning Gaussian mixture models [19], graphbased semi-supervised learning [32], supervised dimensionality reduction [12] and reinforcementlearning [1].
We refer to [20] for a recent review of the literature.
Our work is most closely related to[28], and we compare our findings with the results of that work in Section 2.
We proceed as follows.
Section 2 defines and analyzes the mixed-feature distributionally robustlogistic regression problem, which is solved in Section 3 via column-and-constraint generation.
Wereport numerical results in Section 4.
Auxiliary material and all proofs are relegated to the appendix.
Notation.
We define B = { 0 , 1 } and [ N ] = { 1 , . . .
, N } for N ∈ N .
The set of all probabilitydistributions supported on a set Ξ is denoted by P 0 (Ξ) , while the Dirac distribution placing unitprobability mass on x ∈ R n is denoted by δ x ∈ P 0 ( R n ) .
The indicator function 1 [ E ] attains thevalue 1 ( 0 ) whenever the expression E is (not) satisfied.
Mixed-Feature Distributionally Robust Logistic Regression
Section 2.1 derives an exponential-size convex programming formulation of the distributionallyrobust logistic regression problem that serves as the basis of our analysis.
Section 2.2 compares ourformulation with a naïve model that treats categorical features as continuous ones.
Section 2.3, finally,shows that our formulation, despite its exponential size, can be solved in polynomial time due to thebenign structure of the log-loss function.
It also establishes that in stark contrast to the literature, our formulation does not reduce to a regularized non-robust logistic regression, that is, it constitutes agenuinely novel variant of the logistic regression problem.
Our formulation enjoys strong finite-sample and asymptotic performance guarantees from the literature.
Moreover, despite its exponential size, our model always accommodates worst-case distributionsthat exhibit a desirable sparsity pattern.
We relegate these results to Appendix A.
2.1 Formulation as an Exponential-Size Convex Program
From now on, we consider a mixed-feature data set { ξ i :=
( x i , z
i , y i ) } i ∈ [ N ] with n numericalfeatures x i =
( x i 1 , . . .
, x in ) ∈ R n , m categorical features z
i =
( z i 1 , . . .
, z im ) ∈ C ( k 1 ) × . . .
× C ( k m )and a binary label y i ∈ {− 1 , +1 } .
Here, C ( s ) = { z ∈ B s − 1 : (cid:80) j ∈
[ s − 1]
z j ≤ 1 } for s ∈ N \ { 1 }represents the one-hot encoding of a categorical feature with s possible values; in particular, C (2) =
Bencodes a binary feature.
We denote by C =
C ( k 1 ) × . . .
× C ( k m ) and Ξ = R n × C × {− 1 , +1 } thesupport of the categorical features as well as the data set, respectively, and we let k = k 1 + . . .
+ k m − mbe the number of slopes used for the all categorical features.
If we had access to the true data-generating distribution P 0 ∈ P 0 (Ξ) , we would solve the (non-robust)logistic regression problemminimize β
[Equation]
subject to β = ( β [Equation] [Equation] β C
[Equation]
R n +
[Equation]
+
[Equation]
, where E P 0 denotes the expectation under P 0 , and where the log-loss function l β now takes the form
[Equation]
to account for the presence of categorical features.
Since the distribution P 0 is unknown in practice, theempirical risk minimization problem replaces P 0 with the empirical distribution (cid:98) P N :
= 1 N (cid:80) Ni =
δ ξ ithat places equal probability mass on all observations { ξ i } i ∈ [ N ] .
Standard arguments show thatwhen these observations are i.i.d., the empirical risk minimization problem recovers the logisticregression under P 0 as N −→ ∞ .
In practice, however, data tends to be scarce, and the empiricalrisk minimization problem exhibits an ‘optimism bias’ that is also known as overfitting [6, 17, 25],the error maximization effect of optimization [9, 22] or the optimizer’s curse [31].
Distributionally robust logistic regression combats the aforementioned overfitting phenomenon bysolving the semi-infinite optimization problem
[Equation]
to [Equation] where the ambiguity set B ϵ ( (cid:98) P N ) contains all distributions Q in a (soon to be defined) vicinity of theempirical distribution (cid:98) P N , and where the expectation is taken with respect to Q .
Problem (1) can beinterpreted as a zero-sum game between the decision maker, who chooses a logistic regression modelparameterized by β , and a fictitious adversary that observes β and subsequently chooses the ‘worst’distribution (in terms of the incurred log-loss) from B ϵ ( (cid:98) P N ) .
Contrary to the classical non-robustlogistic regression, problem (1) guarantees to over estimate the log-loss incurred by β under theunknown true distribution P
as long as P 0 is contained in B ϵ ( (cid:98) P N ) .
On the other hand, problem (1)recovers the empirical risk minimization problem when the ambiguity set B ϵ ( (cid:98) P N ) approaches asingleton set that only contains the empirical distribution (cid:98) P N .
Note that problem (1) is convex asthe convexity of the logistic regression objective E Q [ l β ( x , z , y )] is preserved under the supremumoperator.
That said, the problem typically constitutes a semi-infinite program as is comprises finitelymany decision variables
but—if the embedded supremum in the objective is brought to the constraintsvia an epigraph reformulation—infinitely many constraints whenever the ambiguity set harborsinfinitely many distributions.
As such, it is not obvious how the problem can be solved efficiently.
In this paper, we choose as ambiguity set the Wasserstein ball B ϵ ( (cid:98) P N ) :
= { Q ∈ P 0 (Ξ) :
W( Q , (cid:98) P N ) ≤ ϵ } of radius ϵ > 0 that is centered at the empirical distribution (cid:98) P N .
Definition 1 (Wasserstein Distance) .
The type-1 Wasserstein ( Kantorovich-Rubinstein , or earthmover’s ) distance between two distributions P ∈ P 0 (Ξ) and Q ∈ P
(Ξ) is defined as
[Equation]
where ξ = ( x , z , y ) ∈ Ξ and ξ ′
=
( x ′ , z ′ , y ′ )
∈ Ξ , while d ( ξ , ξ ′ ) is the ground metric on Ξ .
The Wasserstein distance can be interpreted as the minimum cost of moving Q to P when d ( ξ , ξ ′ )
isthe cost of moving a unit mass from ξ to ξ ′ .
The Wasserstein radius ϵ thus imposes a budget on thetransportation cost that the adversary can spend on perturbing the empirical distribution (cid:98) P N .
We next define the ground metric d that we use throughout the paper.
Definition 2 (Ground Metric) .
We measure the distance between two data-points ξ =
( x , z , y ) ∈ Ξand ξ ′
=
( x ′ , z ′ , y ′ )
∈ Ξ with z = ( z 1 , . . .
, z m ) and z ′ =
( z ′ 1 , . . .
, z ′ m ) as
[Equation]
3a
[Equation]
where ∥·∥ is any rational norm on R n , κ > 0 and the metric d C on C satisfies
[Equation]
p
[Equation]
[Equation]
Intuitively, the ground metric d measures the distance of the numerical features x and x ′ by the normdistance ∥ x − x ′ ∥ , whereas the distance of the categorical features z and z ′ is measured by (a powerof) the number of discrepancies between z and z ′ .
Likewise, any discrepancy between the labels yand y ′ is accounted for by a constants κ , which allows for scaling between the features and the labels.
We first extend the convex optimization model of [28] for the distributionally robust continuousfeature logistic regression to an exponential conic model that accommodates for categorical features.
Theorem 1 (Exponential Conic Representation) .
The distributionally robust logistic regressionproblem (1) admits the equivalent reformulationminimize
[Equation]
subject to
[Equation]
as a finite-dimensional exponential conic program, where K exp denotes the exponential cone
[Equation]
Problem (4) is finite-dimensional and convex.
Moreover, since exponential conic programs admit aself-concordant barrier function, they can be solved in polynomial time relative to their input size[26].
Due to the presence of categorical features in our regression problem, however, problem (4)comprises exponentially many variables and constraints, and a naïve solution of (4) would thereforerequire an exponential amount of time.
The exponential size of problem (4) renders our formulationfundamentally different from the model of [28], and it significantly complicates the solution.
2.2 Comparison with Continuous-Feature Distributionally Robust Logistic Regression
Our formulation (4) of the mixed-feature distributionally robust logistic regression (1) accounts forcategorical features z ∈ C at the expense of an exponential number of variables and constraints.
It is therefore tempting to treat the categorical features as continuous ones and directly apply thecontinuous-feature-only reformulation of the distributionally robust logistic regression (1) proposedby [28].
In the following, we argue that such a reformulation would hedge against nonsensical worstcase distributions that in turn lead to overly conservative regression models.
To see this, consider astylized setting where the data set ( z i , y i )
i ∈
[ N ] comprises a single binary feature z i ∈ {− 1 , +1 } thatimpacts the label y i ∈ {− 1 , +1 } via the logistic model
[Equation]
with β = 1 .
We attempt to recover this logistic model in two ways:
Mixed-feature model.
We employ our distributionally robust logistic regression (1) with asingle categorical feature, using p = 1 and κ = 1 in the ground metric ( cf.
Definition 2).2.
Continuous -feature model.
We employ a variant of problem (1) that treats the categoricalfeature
z i as a continuous one.
In order to make the ambiguity set directly comparable tothat of our mixed-feature model, we choose as ground metric
[Equation]
that is, the feature distances
[ z ̸ = z ′ ] in the mixed-feature case and 12 |
z − z ′ | in thecontinuous-feature case both attain the value 0 ( 1 ) for equal (different) feature values.
Figure 1 (left) compares the mean values (dashed lines) as well as the 15% and 85% quantiles (shadedregions) of β for the standard logistic regression, our mixed-feature as well as the continuous-featuremodel as a function of the size N of the data set.
For both distributionally robust models, weemployed the same Wasserstein radius ϵ ∝ 1 / √ N , which is motivated by the finite sample guaranteepresented in the next section ( cf.
Theorem 6).
The figure shows that the continuous-feature modelexcessively shrinks its estimates of β , thus leading to overly conservative results.
In fact, the trueparameter value β = 1 consistently lies outside the confidence region of the continuous-featuremodel, independent of the sample size N .
This is due to the fact that the continuous-feature modelaccounts for nonsensical worst-case distributions under which the categorical feature can take valuesoutside its domain {− 1 , +1 } , see Figure 1 (right).
In fact, the continuous-feature model placesnon-zero probability on an extreme scenario under which the binary feature z ∈ {− 1 , +1 } attains thevalue 15 , 312 .
Our mixed-feature model, on the other hand, restricts the worst-case distribution to thedomain of the categorical feature and thus hedges against realistic distributions only.
We note that in practice, the radius ϵ of the Wasserstein ball will be chosen via cross-validation( cf.
Section 4), in which case our mixed-feature model reliably outperforms the classical logisticregression on standard benchmark instances.
While one may argue that the impact of nonsensicalworst-case distributions in the continuous-feature model is alleviated by the cross-validated radii,
Figure 1 offers strong theoretical and practical reasons against the use of a continuous-feature modelfor categorical or mixed-feature problems: The continuous-feature model hedges against a clumsilyperturbed worst-case distribution that introduces a pronounced bias in the estimation.
Moreover,
Section 4 will show that the mixed-feature model, despite its exponential size, can be solved quicklyand reliably with our novel column-and-constraint approach from Section 3.
2.3 Complexity Analysis
We first show that despite its exponential size, the distributionally robust logistic regression problem (1) admits a polynomial-time solution.
Key to this perhaps surprising finding is the shape of theloss function: while problem (1) is strongly NP-hard (and thus unlikely to admit a polynomial-timesolution scheme) for generic loss functions, it can be solved in polynomial time for the log-lossfunction l β employed in logistic regression.
Theorem 2 (Complexity of the Distributionally Robust Logistic Regression (1)) .
(i) For generic loss functions l β , problem (1) is strongly NP-hard even if n = 0 and N = 1 .
(ii)
For the loss function l β ( x , z , y ) =
log (cid:0) 1 + exp (cid:2) − y · (cid:0) β 0 + β N ⊤ x + β C ⊤ z
(cid:1)(cid:3)(cid:1) and the ground metric of Definition 2, problem (1) can be solved to δ -accuracy in polynomial time.
Recall that an optimization problem is solved to δ -accuracy if a δ -suboptimal solution is identified thatsatisfies all constraints modulo a violation of at most δ .
The consideration of δ -accurate solutions isstandard in the numerical solution of nonlinear programs where an optimal solution may be irrational.
A by now well-known result shows that when m =
(no categorical features), the distributionallyrobust logistic regression problem (1) reduces to a classical logistic regression with an additionalregularization term ∥ β x ∥ ∗ in the objective function when the output label weight κ in Definition 2approaches ∞ [28, 29].
We next show that this reduction to a classical regularized logistic regressionno longer holds in our problem setting when categorical features are present.
Theorem 3 (Absence of a Reformulation as a Regularized Problem) .
Even when the output labelweight κ approaches ∞ in the distributionally robust logistic regression (1) , problem (1) does notadmit an equivalent reformulationminimize
[Equation]
subject to β = ( β 0 , β N , β C )
∈ R n + k +1 , as a classical regularized logistic regression for any regularizer R : R n + k +1 → R .
Note that Theorem 3 does not only preclude the existence of a specific regularizer, but it excludes theexistence of any regularizer, no matter how complex its dependence on β might be.
We are not awareof any prior results of this form in the literature.
Column-and-Constraint Solution Scheme
Our distributionally robust logistic regression problem (4) comprises exponentially many variablesand constraints, which renders its solution as a monolithic exponential conic program challenging.
Instead, Algorithm 1 employs a column-and-constraint generation scheme that alternates between(i) the solution of relaxations of problem (4) that omit most of its variables and constraints and (ii)adding those variables and constraints that promise to maximally tighten the relaxations.
Theorem 4. Algorithm 1 solves problem (4) in finitely many iterations.
Moreover, LB t and UB tconstitute monotone sequences of lower and upper bounds on the optimal value of problem (4) .
A key step in Algorithm 1 is the identification of most violated indices ( i, z ) ∈ [ N ] × C of the firstand second constraint set in the reduced distributionally robust regression problem (4) for a fixed
Algorithm 1 Column-and-Constraint Generation Scheme for Problem (4).
Set LB 0 = −∞ and UB 0 = + ∞
Choose (possibly empty) subsets W + , W − ⊆
[ N ] × C while LB t ̸ = UB t do
Let θ ⋆ be the optimal value and ( β , λ, s , u ± , v ± ) be a minimizer of problem (4) where:
• the first constraint set is only enforced for all ( i, z ) ∈ W + ; • the second constraint set is only enforced for all ( i, z ) ∈ W − ; • we only include those ( u + i, z , v
+ i, z ) for which ( i, z ) ∈ W + ; • we only include those ( u − i, z , v − i, z ) for which ( i, z ) ∈ W − .
Identify a most violated index ( i + , z + ) of the first constraint set in (4) and add it to W +
Identify a most violated index ( i − , z − ) of the second constraint set in (4) and add it to W −
Let ϑ + and ϑ − be the violations of ( i + , z + ) and ( i − , z − ) , respectively
Update LB t = θ ⋆ and UB t = θ ⋆ + log(1 + max { ϑ + , ϑ − } )
Update t = t +end while
Algorithm 2 Identification of Most Violated Constraints in the Reduced Problem (4).
for j ∈ { 1 , . . .
, m } do
Find a feature value z ⋆j that minimizes y i · β C ,j ⊤ z j across all z j ∈ C ( k j ) \ { z ij } end for
Let π : [ m ] → [ m ] be an ordering
such that y iπ ( j ) · β C ,π ( j ) ⊤ ( z ⋆π ( j ) − z iπ ( j ) ) ≤ y iπ ( j ′ ) · β C ,π ( j ′ )
⊤ ( z ⋆π ( j ′ )
− z iπ ( j ′ ) ) ∀ 1 ≤ j ≤ j ′ ≤ m
Set W = ∅ for δ ∈ { 0 , 1 , . . .
, m } do
Update W = W ∪ { z } , where z j
= z
⋆j if π ( j ) ≤ δ and z j = z ij otherwise end for
Determine the most violated constraint from the candidate set W solution ( β , λ, s , u ± , v ± ) .
For the first constraint set in (4), the identification of such indices requiresfor each data point i ∈
[ N ] the solution of the combinatorial problemmaximize z
[Equation]
subject to
[Equation]
where an optimal value greater than 1 corresponds to a violated constraint; an analogous problem canbe defined for the second constraint set in (4).
Despite its combinatorial nature, the above problemcan be solved efficiently by means of Algorithm 2.
Theorem 5.
Algorithm 2 identifies a most violated constraint for a given data-point i ∈ [ N ] in time
O ( k + n + m 2 ) .
Algorithm 2 first determines for each categorical feature z j , j = 1 , . . .
, m , the feature value z ⋆j ∈
C ( k j ) \ { z ij } that, ceteris paribus , contributes to a maximal constraint violation in time O ( k ) .
Thesubsequent step sorts the different features j = 1 , . . .
, m in descending order of their contribution toconstraint violations in time O
( m log m ) .
The last step of Algorithm 2, finally, constructs the mostviolated constraint z across those in which δ = 0 , . . .
, m categorical feature values deviate from z iand subsequently picks the most violated constraint of these in time O ( n + m 2 ) .
We note that theoverall complexity of O ( k + n + m 2 ) can be reduced to O ( k + n + m log m ) by a clever use of datastructures in the final step; for ease of exposition, we omit the details.
The intuition behind Algorithm 2 is as follows.
In the above combinatorial optimization problem,the decision variables z ∈ C only appear in (monotone transformations of) the terms − y i β C ⊤ z
and− λd C ( z , z i ) . Moreover, the expression [ d C ( z , z i )] p can only attain one of the values δ ∈ { 0 , . . . , m }( cf.
Definition 2). Conditioning on each possible value of δ , which records the number of categorical features along which z and z i disagree, we can therefore maximize the constraint violation bymaximizing − y i β C ⊤ z along all z ∈ C that differ from z i in exactly δ categorical features.
Finally,since − y i β C ⊤ z is linearly separable, that is, − y i
β C ⊤ z =
− y i
β C , 1 ⊤ z 1 − . . .
− y i
β C ,m ⊤ z m , wecan choose the categorical features along which z
and z i disagree iteratively by considering eachexpression − y i
β C ⊤ z separately and choosing the disagreeing feature values greedily based on theirindividual contribution to the overall constraint violation.  Numerical Results Section 4.1 first compares the runtimes of our column-and-constraint scheme from Section 3 withthose of solving the distributionally robust logistic regression problem (4) naïvely as a monolithicexponential conic program.
We subsequently compare the classification performance of problem (1)with those of a classical unregularized and regularized logistic regression on standard benchmarkinstances with categorical features (Section 4.2) and mixed features (Section 4.3).
All algorithms were implemented in Julia [5] (MIT license) and executed on Intel Xeon 2.66GHzprocessors with 8GB memory in single-core mode.
We use MOSEK 9.3 [24] (commercial) to solveall exponential conic programs through JuMP [11] (MPL2 License).
All source codes and detailedresults are available on GitHub (URL withheld to maintain anonymity during the review process).
4.1 Runtime Comparison with Monolithic Formulation
We first compare the computation times of our column-and-constraint scheme from Section 3 withthose of solving the distributionally robust logistic regression problem (4) naïvely as a monolithicexponential conic program.
To this end, we randomly generate synthetic logistic regression instanceswith varying numbers N of data points and m of binary features.
While Figure 2 (left) shows thatboth approaches scale similarly in the number N of data points, Figure 2 (middle) reveals that thesolution of the monolithic formulation scales exponentially in the number m of binary features.
Incontrast, our column-and-constraint scheme scales gracefully in both the number N of data pointsand the number m of binary features ( cf.
Figure 2, right).
A similar behavior can be observed withmore general, non-binary categorical features; we omit the results due to space constraints.
4.2 Performance on Categorical-Feature Instances
We next compare the classification performance (in terms of the out-of-sample classification error) ofour unregularized (‘DRO’) and Lasso-regularized (‘r-DRO’) distributionally robust logistic regressionproblem (1) with those of a classical unregularized (‘LR’) and Lasso-regularized (‘r-LR’) logisticregression on the 18 most popular UCI data sets that only contain categorical features [10] (varyinglicenses).
Instances with multiple output labels are indicated with a star; we convert them intoinstances with binary output labels by distinguishing between the majority class vs. all other classes.
The instances vary in the number N of data points, the number m of categorical features as well as,accordingly, the number k of slopes considered in the one-hot encoding of the categorical features( cf.
Section 2.1).
All results are reported as medians over 20 random training set-test set splits [Table] [Table]
The radius ϵ ∈ { 0 , 10 − 5 , . . .
, 10 − 4 , . .
, 1 } of the Wasserstein ball as well as the Lassopenalty γ ∈ { 0 , 12 · 10 − 5 , . .
, 12 · 10 − 4 , . .
. , 12 } are selected via 5-fold cross-validation.
We considertwo variants of our distributionally robust logistic regression that employ a different output labelweight ( κ = 1 vs. κ = m ) in the ground metric ( cf.
Definition 2).
The results are reported in Table 1.
The table shows that for the unregularized model, both of our distributionally robust logisticregression achieve the lowest classification error in 78% of the instances, whereas the classical logisticregression achieves the lowest classification error in 61% of the instances.
For the regularized model,the results remain at 78% (our model) vs. 61% (classical logistic regression).
Overall, the non-robustmodels failed to achieve the lowest classification error in three of the instances, whereas the robustmodels missed the lowest classification error in only one of the instances. 4.3 Performance on Mixed-Feature Instances
We repeat the experiment of the previous subsection for the five most popular mixed-feature instancesof the UCI data set [10]. The results are reported in Table 2.
The conclusions are qualitatively similarto those of Section 4.3. We highlight, however, that now only two of the lowest classification errorsare achieved by one of the non-robust models, whereas the lowest classification error in each instanceis obtained by at least one of the robust models.
Conclusions
We proposed a new distributionally robust mixed-feature logistic regression model where the proximity between the empirical distribution and the unknown true data-generating distribution is measuredby the popular Wasserstein distance.
Despite its exponential-size formulation, we prove that theunderlying optimization problem can be solved in polynomial time, and we develop a practicallyefficient column-and-constraint generation scheme for its solution.
The promising performance of ourmodel is demonstrated in numerical experiments on standard benchmark instances.
We note that ourcolumn-and-constraint scheme readily extends to other distributionally robust mixed-feature machinelearning models such as linear regression, support vector machines and decision trees.
Applying ouralgorithm in those settings is a promising direction for future research.