Wasserstein Logistic Regression with Mixed Features
Abstract
Recent work has leveraged the popular distributionally robust optimization paradigm to combat overfitting in classical logistic regression.
While the resulting classification scheme displays a promising performance in numerical experiments, it is inherently limited to numerical features.
In this paper, we show that distributionally robust logistic regression with mixed ( i.e. , numerical and categorical) features, despite amounting to an optimization problem of exponential size, admits a polynomial-time solution scheme.
We subsequently develop a practically efficient column-and-constraint approach that solves the problem as a sequence of polynomial-time solvable exponential conic programs.
Our model retains many of the desirable theoretical features of previous works, but—in contrast to the literature—it does not admit an equivalent representation as a regularized logistic regression, that is, it represents a genuinely novel variant of logistic regression.
We show that our method outperforms both the unregularized and the regularized logistic regression on categorical as well as mixed-feature benchmark instances.
Introduction
Consider a data set ( x i , y i )
Ni =1 with feature vectors x i and associated binary labels y i ∈ {− 1 , 1 } .
Classical logistic regression assumes that the labels depend probabilistically on the features via
Prob ( y | x ) = (cid:2) 1 + exp( − y · [ β 0 + β ⊤ x ]) (cid:3) − 1 , where the parameters ( β 0 , β ) ∈ R 1+ n are estimated from the empirical risk minimization problem minimize ( β 0 , β )
N
N (cid:88) i
= l β ( x i , y i ) subject to ( β 0 , β ) ∈ R n + with the log-loss function l β ( x , y ) :
= log (cid:0) 1 + exp (cid:0) − y · [ β 0 + β ⊤ x ] (cid:1)(cid:1) .
Its compelling performance across many domains, the availability of mature and computationally efficient algorithms as well as its interpretability have all contributed to the widespread adoption of logistic regression [6, 18, 27].
Similar to other machine learning models, logistic regression is prone to overfitting, especially when the number of training samples is small relative to the number of considered features.
Moreover, logistic regression can be sensitive to erroneous feature and label values as well as distribution shifts under which the training and test sets stem from different distributions.
In recent years, distributionally robust (DR) optimization [3, 4] has been proposed to simultaneously address these challenges.
To this end, the DR optimization paradigm models a machine learning task as a zero-sum game between the decision maker, who seeks to obtain the most accurate model ( e.g. , the parameter vector β in a logistic regression), and a fictitious adversary who observes the decision maker’s model and subsequently selects the worst data-generating distribution in the vicinity of the empirical distribution formed from the available training data.
In this context, the similarity of distributions is commonly measured in terms of moment bounds [9, 39], ϕ -divergences such as the Kullback-Leibler divergence [19, 22] or the Wasserstein distance [15, 25].
It has been observed that in many cases, the resulting
DR machine learning models are equivalent to regularized versions of the same models, in which case distributionally robustness provides a new perspective on regularizers that are often selected ad hoc [21, 41].
Most importantly, DR machine learning models often admit computationally efficient reformulations as finite-dimensional convex optimization problems that can be solved in polynomial time with off-the-shelf solvers.
In this work we study a DR variant of the logistic regression where the distance between the empirical distribution and the unknown true data-generating distribution is measured by the popular
Wasserstein (also known as Kantorovich-Rubinstein or earth mover’s) distance [15, 25].
This problem has been first studied by [31], who show that the resulting DR logistic regression problem admits an equivalent reformulation as a polynomial-size convex optimization problem if all features are numerical .
Applying similar techniques to the mixed-feature logistic regression problem, which appears to be most prevalent in practice, would result in an exponential-size convex optimization problem whose naïve solution with the methods of [31] does not scale to interesting problem sizes ( cf.
Section 4).
Our contributions may be summarized as follows.
(i)
On the theoretical side , we show that the complexity of DR mixed-feature regression crucially relies on the selected loss function.
In particular, for the log-loss function employed in logistic regression, the problem—despite its natural representation as an exponential-size convex optimization problem—admits a polynomial-time solution scheme.
We also show that in stark contrast to earlier variants of the problem, our mixed-feature regression does not admit an equivalent representation as a regularized problem.
This provides compelling evidence that the DR logistic regression problem with mixed features is fundamentally different to the DR problem with only numerical features.
(ii)
On the computational side , we propose a column-and-constraint scheme that solves the DR mixed-feature logistic regression as a sequence of polynomial-time solvable exponential conic programs.
We show that the key step of our procedure, the identification of the most violated constraint, can be implemented efficiently for a broad range of metrics, despite its natural representation as a combinatorial optimization problem.
Indeed, identifying the most violated constraint by brute force is out of the question, while the standard approach [43] of solving the most violated constraint problem would have an unacceptably high runtime.
(iii)
On the numerical side , we show that our column-and-constraint scheme drastically reduces computation times over a naïve monolithic implementation of the regression problem.
We also show that our model performs favorably on standard categorical and mixed-feature benchmark instances when compared against classical and regularized logistic regression.
The literature on DR machine learning under the Wasserstein distance is vast and rapidly growing.
Recent works have considered, among others, the use of Wasserstein DR models in multi-label learning [14], generative adversarial networks [2, 8] and the generation of adversarial examples [40], density estimation [38] and learning Gaussian mixture models [20], graph-based semi-supervised learning [35], supervised dimensionality reduction [13] and reinforcement learning [1].
DR has also been found to help alleviate problems with overfitting [24, 21], label uncertainty [21] and distribution shifts [37, 36].
We refer to [21] for a recent review of the literature.
Our work is most closely related to [31], and we compare our findings with the results of that work in Section 2.
We proceed as follows.
Section 2 defines and analyzes the mixed-feature DR logistic regression problem, which is solved in Section 3 via column-and-constraint generation.
We report numerical results in Section 4.
Auxiliary material and all proofs are relegated to the appendix.
Notation.
We define B = { 0 , 1 } and [ N ] = { 1 , . . .
, N } for N ∈ N .
The set of all probability distributions supported on a set Ξ is denoted by P 0 (Ξ) , while the Dirac distribution placing unit probability mass on x ∈ R n is denoted by δ x ∈ P 0 ( R n ) .
The indicator function 1 [ E ] attains the value 1 ( 0 ) whenever the expression E is (not) satisfied.
Finally, we use Roman d to denote differentials and to distinguish them from the d used to denote distances.
Mixed-Feature DR Logistic Regression
Section 2.1 derives an exponential-size convex programming formulation of the DR logistic regression problem that serves as the basis of our analysis.
Section 2.2 compares our formulation with a naïve model that treats categorical features as continuous ones.
Section 2.3, finally, shows that our formulation, despite its exponential size, can be solved in polynomial time due to the benign structure of the log-loss function.
It also establishes that in stark contrast to the literature, our formulation does not reduce to a regularized non-robust logistic regression, that is, it constitutes a genuinely novel variant of the logistic regression problem.
Our formulation enjoys strong finite-sample and asymptotic performance guarantees from the literature.
Moreover, despite its exponential size, our model always accommodates worst-case distributions that exhibit a desirable sparsity pattern.
We relegate these results to Appendix A.
2.1 Formulation as an Exponential-Size Convex Program
From now on, we consider a mixed-feature data set { ξ i :=
( x i , z
i , y i ) } i ∈ [ N ] with n numerical features x i =
( x i 1 , . . .
, x in ) ∈ R n , m categorical features z
i =
( z i 1 , . . .
, z im ) ∈ C ( k 1 ) × . . .
× C ( k m ) and a binary label y i ∈ {− 1 , +1 } .
Here, C ( s ) = { z ∈ B s − 1 : (cid:80) j ∈
[ s − 1]
z j ≤ 1 } for s ∈ N \ { 1 } represents the one-hot encoding of a categorical feature with s possible values; in particular, C (2) =
B encodes a binary feature.
We denote by C =
C ( k 1 ) × . . .
× C ( k m ) and Ξ = R n × C × {− 1 , +1 } the support of the categorical features as well as the data set, respectively, and we let k = k 1 + . . .
+ k m − m be the number of slopes used for the categorical features.
If we had access to the true data-generating distribution P 0 ∈ P 0 (Ξ) , we would solve the (non-robust) logistic regression problem
[Equation]
subject
[Equation]
where E P 0 denotes the expectation under P 0 , and where the log-loss function l β now takes the form
[Equation]
to account for the presence of categorical features.
Since the distribution P 0 is unknown in practice, the empirical risk minimization problem replaces P 0 with the empirical distribution (cid:98) P N :
= 1 N (cid:80) Ni =
δ ξ i that places equal probability mass on all observations { ξ i } i ∈ [ N ] .
Standard arguments show that when these observations are i.i.d., the empirical risk minimization problem recovers the logistic regression under P 0 as N −→ ∞ .
In practice, however, data tends to be scarce, and the empirical risk minimization problem exhibits an ‘optimism bias’ that is also known as overfitting [6, 18, 27], the error maximization effect of optimization [10, 23] or the optimizer’s curse [34].
DR logistic regression combats the aforementioned overfitting phenomenon by solving the semiinfinite optimization problem
[Equation]
where the ambiguity set B ϵ ( (cid:98) P N ) contains all distributions Q in a (soon to be defined) vicinity of the empirical distribution (cid:98) P N , and where the expectation is taken with respect to Q .
Problem (1) can be interpreted as a zero-sum game between the decision maker, who chooses a logistic regression model parameterized by β , and a fictitious adversary that observes β and subsequently chooses the ‘worst’ distribution (in terms of the incurred log-loss) from B ϵ ( (cid:98) P N ) .
Contrary to the classical non-robust logistic regression, problem (1) guarantees to over estimate the log-loss incurred by β under the unknown true distribution P
as long as P 0 is contained in B ϵ ( (cid:98) P N ) .
On the other hand, problem (1) recovers the empirical risk minimization problem when the ambiguity set B ϵ ( (cid:98) P N ) approaches a singleton set that only contains the empirical distribution (cid:98) P N .
Note that problem (1) is convex as the convexity of the logistic regression objective E Q [ l β ( x , z , y )] is preserved under the supremum operator.
That said, the problem typically constitutes a semi-infinite program as it comprises finitely many decision variables but—if the embedded supremum in the objective is brought to the constraints via an epigraph reformulation—infinitely many constraints whenever the ambiguity set harbors infinitely many distributions.
As such, it is not obvious how the problem can be solved efficiently.
In this paper, we choose as ambiguity set the Wasserstein ball B ϵ ( (cid:98) P N ) :
= { Q ∈ P 0 (Ξ) :
W( Q , (cid:98) P N ) ≤ ϵ } of radius ϵ > 0 that is centered at the empirical distribution (cid:98) P N .
Definition 1 (Wasserstein Distance) .
The type-1 Wasserstein ( Kantorovich-Rubinstein , or earth mover’s ) distance between two distributions P ∈ P 0 (Ξ) and Q ∈ P
(Ξ) is defined as
[Equation]
where ξ = ( x , z , y ) ∈ Ξ and ξ ′
=
( x ′ , z ′ , y ′ )
∈ Ξ , while d ( ξ , ξ ′ ) is the ground metric on Ξ .
The Wasserstein distance can be interpreted as the minimum cost of moving Q to P when d ( ξ , ξ ′ ) is the cost of moving a unit mass from ξ to ξ ′ .
The Wasserstein radius ϵ thus imposes a budget on the transportation cost that the adversary can spend on perturbing the empirical distribution (cid:98) P N .
We next define the ground metric d that we use throughout the paper.
Definition 2 (Ground Metric) .
We measure the distance between two data-points ξ =
( x , z , y ) ∈ Ξ and ξ ′
=
( x ′ , z ′ , y ′ )
∈ Ξ with z = ( z 1 , . . .
, z m ) and z ′ =
( z ′ 1 , . . .
, z ′ m ) as
[Equation]
where ∥·∥ is any rational norm on R n , κ > 0 and the metric d C on C satisfies
[Equation]
Intuitively, the ground metric d measures the distance of the numerical features x and x ′ by the norm distance ∥ x − x ′ ∥ , whereas the distance of the categorical features z and z ′ is measured by (a power of) the number of discrepancies between z and z ′ .
Likewise, any discrepancy between the labels y and y ′ is accounted for by a constant κ , which allows for scaling between the features and the labels.
We first extend the convex optimization model of [31] for the DR continuous-feature logistic regression to an exponential conic model that accommodates for categorical features.
Theorem 1 (Exponential Conic Representation) .
The DR logistic regression problem (1) admits the equivalent reformulation minimize β ,λ, s , u ± , v ± λϵ + 1 N (cid:88)
i ∈
[ N ] s i subject to u
+ i, z + v
+ i, z ≤( u + i, z , 1 , − s i − λd C
( z , z i ))
∈ K exp ( v + i, z , 1 , − y i β N ⊤ x
i − y i β C ⊤ z − y i β 0 − s i − λd C
( z , z i ))
∈ K exp   ∀ i ∈
[ N ] , ∀ z ∈ C u − i, z
+ v − i, z ≤( u − i, z , 1 , − s i − λd C ( z , z i ) − λκ ) ∈ K exp ( v − i, z , 1 , y i β N ⊤
x
i
+ y
i β C ⊤
z
+ y i β 0 − s i − λd C
( z , z i ) − λκ ) ∈ K exp  
[Equation]
as a finite-dimensional exponential conic program, where K exp denotes the exponential cone
K exp := cl ( { ( a, b, c ) : a ≥ b · exp( c/b ) , a > 0 , b > 0 } ) ⊂ R 3 .
The s i ’s and λ that appear in (4) are dual variables that arise from dual of the inner ( i.e. , sup ) problem in (1).
For each z ∈ C and i ∈ [ N ] , ( u + i, z , v
+ i, z , u − i, z , v − i, z ) is a vector of auxiliary variables that we use to model softplus constraints that arise from our intermediate analysis of problem (1) (Appendix E).
Problem (4) is finite-dimensional and convex.
Moreover, since exponential conic programs admit a self-concordant barrier function, they can be solved in polynomial time relative to their input size [28].
Due to the presence of categorical features in our regression problem, however, problem (4) comprises exponentially many variables and constraints, and a naïve solution of (4) would therefore require an exponential amount of time.
The exponential size of problem (4) renders our formulation fundamentally different from the model of [31], and it significantly complicates the solution.
2.2 Comparison with Continuous-Feature DR Logistic Regression
Our formulation (4) of the mixed-feature DR logistic regression (1) accounts for categorical features z ∈ C at the expense of an exponential number of variables and constraints.
It is therefore tempting to treat the categorical features as continuous ones and directly apply the continuous-feature-only reformulation of the DR logistic regression (1) proposed by [31].
In the following, we argue that such a reformulation would hedge against nonsensical worst-case distributions that in turn lead to overly conservative regression models.
To see this, consider a stylized setting where the data set ( z i , y i )
i ∈
[ N ] comprises a single binary feature z i ∈ {− 1 , +1 } that impacts the label y i ∈ {− 1 , +1 } via the logistic model
[Equation]
with β = 1 .
(While we use z i ∈ { 0 , 1 } in the other sections of the paper, the analysis and results of this subsection and Appendix B assumed z i ∈ {− 1 , +1 } as this made comparisons with [31] easier.)
We attempt to recover this logistic model in two ways:
Mixed-feature model.
We employ our DR logistic regression (1) with a single categorical feature, using p = 1 and κ = 1 in the ground metric ( cf.
Definition 2).
Continuous-feature model.
We employ a variant of problem (1) that treats the categorical feature
z i as a continuous one.
We choose as ground metric
[Equation]
We therefore use the feature distances
[ z ̸ = z ′ ] in the mixed-feature case and 1 2 | z − z ′ | in the continuous-feature case.
Note they both attain the value 0 ( 1 ) for equal (different) feature values.
Figure 1 (left) compares the mean values (dashed lines) as well as the 15% and 85% quantiles (shaded regions) of β for the standard logistic regression, our mixed-feature as well as the continuous-feature model as a function of the size N of the data set.
For both DR models, we employed the same
Wasserstein radius ϵ ∝ 1 / √ N , which is motivated by the finite sample guarantee presented in the next section ( cf.
Theorem 6).
The figure shows that the continuous-feature model excessively shrinks its estimates of β , thus leading to overly conservative results.
In fact, the true parameter value β = consistently lies outside the confidence region of the continuous-feature model, independent of the sample size N .
This is due to the fact that the continuous-feature model accounts for nonsensical worst-case distributions under which the categorical feature can take values outside its domain {− 1 , +1 } , see Figure 1 (right).
In fact, the continuous-feature model places non-zero probability on an extreme scenario under which the binary feature z ∈ {− 1 , +1 } attains the value 15 , 312 .
(In
Appendix B we explain why this value of 15 , 312 arises.)
Our mixed-feature model, on the other hand, restricts the worst-case distribution to the domain of the categorical feature and thus hedges against realistic distributions only.
We note that in practice, the radius ϵ of the Wasserstein ball will be chosen via cross-validation ( cf.
Section 4), in which case our mixed-feature model reliably outperforms the classical logistic regression on standard benchmark instances.
While one may argue that the impact of nonsensical worst-case distributions in the continuous-feature model is alleviated by the cross-validated radii,
Figure 1 offers strong theoretical and practical reasons against the use of a continuous-feature model for categorical or mixed-feature problems: The continuous-feature model hedges against a clumsily perturbed worst-case distribution that introduces a pronounced bias in the estimation.
We also note that another possible approach would be to use the continuous-feature model but to restrict the support of binary features to [ − 1 , 1] .
Unfortunately, for the log-loss function no tractable reformulation with support constraints appears to be known; see [32] .
Moreover, Section 4 will show that the mixed-feature model, despite its exponential size, can be solved quickly and reliably with our novel column-and-constraint approach from Section 3.
2.3 Complexity Analysis
We first show that despite its exponential size, the DR logistic regression problem (1) admits a polynomial-time solution.
Key to this perhaps surprising finding is the shape of the loss function: while problem (1) is strongly NP-hard (and thus unlikely to admit a polynomial-time solution scheme) for generic loss functions, it can be solved in polynomial time for the log-loss function l β employed in logistic regression.
Theorem 2 (Complexity of the DR Logistic Regression (1)) .
(i) For generic loss functions l β , problem (1) is strongly NP-hard even if n = 0 and N = 1 .
(ii)
For the loss function l β ( x , z , y ) =
log (cid:0) 1 + exp (cid:2) − y · (cid:0) β 0 + β N ⊤ x + β C ⊤ z
(cid:1)(cid:3)(cid:1) and the ground metric of Definition 2, problem (1) can be solved to δ -accuracy in polynomial time
Recall that an optimization problem is solved to δ -accuracy if a δ -suboptimal solution is identified that satisfies all constraints modulo a violation of at most δ .
The consideration of δ -accurate solutions is standard in the numerical solution of nonlinear programs where an optimal solution may be irrational.
A by now well-known result shows that when m =
(no categorical features), the DR logistic regression problem (1) reduces to a classical logistic regression with an additional regularization term ∥ β x ∥ ∗ in the objective function when the output label weight κ in Definition 2 approaches ∞ [31, 32].
We next show that this reduction to a classical regularized logistic regression no longer holds in our problem setting when categorical features are present.
Theorem 3 (Absence of a Reformulation as a Regularized Problem) .
Even when the output label weight κ approaches ∞ in the DR logistic regression (1) , problem (1) does not admit an equivalent reformulation
[Equation]
as a classical regularized logistic regression for any regularizer R : R 1+ n + k
→ R .
Note that Theorem 3 does not only preclude the existence of a specific regularizer, but it excludes the existence of any regularizer, no matter how complex its dependence on β might be.
We are not aware of any prior results of this form in the literature.
Some insight for the result in Theorem 3 may be found via the special case we consider in its proof in Appendix E and in Remark 2 that follows it.
Algorithm 1 Column-and-Constraint Generation Scheme for Problem (4).
Set LB 0 = −∞ and UB 0 = + ∞
Choose (possibly empty) subsets W + , W − ⊆
[ N ] × C while LB t ̸ = UB t do
Let θ ⋆ be the optimal value and ( β , λ, s , u ± , v ± ) be a minimizer of problem (4) where:
• the first constraint set is only enforced for all ( i, z ) ∈ W + ; • the second constraint set is only enforced for all ( i, z ) ∈ W − ; • we only include those ( u + i, z , v
+ i, z ) for which ( i, z ) ∈ W + ; • we only include those ( u − i, z , v − i, z ) for which ( i, z ) ∈ W − .
Identify a most violated index ( i + , z + ) of the first constraint set in (4) and add it to W +
Identify a most violated index ( i − , z − ) of the second constraint set in (4) and add it to W −
Let ϑ + and ϑ − be the violations of ( i + , z + ) and ( i − , z − ) , respectively
Update LB t = θ ⋆ and UB t = min { UB t − 1 , θ ⋆ + log(1 + max { ϑ + , ϑ − } ) }
Update t = t +end while
Algorithm 2 Identification of Most Violated Constraints in the Reduced Problem (4).
for j ∈ { 1 , . . .
, m } do
Find a feature value z ⋆j that minimizes y i · β C ,j ⊤ z j across all z j ∈ C ( k j ) \ { z ij } end for
Let π : [ m ] → [ m ] be an ordering such that y iπ ( j ) · β C ,π ( j ) ⊤ ( z
[Equation]
( j ) − z iπ ( j ) )
≤ y iπ ( j ′ ) · β C ,π ( j ′ )
⊤ ( z ⋆π ( j ′ )
− z iπ ( j ′ ) ) ∀ 1 ≤ j ≤ j ′ ≤ m
Set W = ∅ for δ ∈ { 0 , 1 , . . .
, m } do
Update W = W ∪ { z
where z j = z ⋆j if π ( j ) ≤ δ and z j = z ij otherwise end for Determine the most violated constraint from the candidate set W
Column-and-Constraint Solution Scheme
Our DR logistic regression problem (4) comprises exponentially many variables and constraints, which renders its solution as a monolithic exponential conic program challenging.
Instead, Algorithm 1 employs a column-and-constraint generation scheme, e.g. , [43], that alternates between (i) the solution of relaxations of problem (4) that omit most of its variables and constraints and (ii) adding those variables and constraints that promise to maximally tighten the relaxations.
Theorem 4.
Algorithm 1 solves problem (4) in finitely many iterations.
Moreover, LB t and UB t constitute monotone sequences of lower and upper bounds on the optimal value of problem (4) .
A key step in Algorithm 1 is the identification of most violated indices ( i, z ) ∈ [ N ] × C of the first and second constraint set in the reduced DR regression problem (4) for a fixed solution ( β , λ, s , u ± , v ± ) .
For the first constraint set in (4), the identification of such indices requires for each data point
i ∈
[ N ] the solution of the combinatorial problem maximize z
[Equation]
subject to z
[Equation]
(5) where an optimal value greater than 1 corresponds to a violated constraint; an analogous problem can be defined for the second constraint set in (4).
Despite its combinatorial nature, the above problem can be solved efficiently by means of Algorithm 2.
Theorem 5.
Algorithm 2 identifies a most violated constraint for a given data-point i ∈ [ N ] in time
O ( k + n + m 2 ) .
Algorithm 2 first determines for each categorical feature z j , j = 1 , . . .
, m , the feature value z ⋆j ∈
C ( k j ) \ { z ij } that, ceteris paribus , contributes to a maximal constraint violation in time O ( k ) .
The subsequent step sorts the different features j = 1 , . . .
, m in descending order of their contribution to constraint violations in time O
( m log m ) .
The last step of Algorithm 2, finally, constructs the most violated constraint z across those in which δ = 0 , . . .
, m categorical feature values deviate from z i and subsequently picks the most violated constraint of these in time O ( n + m 2 ) .
We note that the overall complexity of O ( k + n + m 2 ) can be reduced to O ( k + n + m log m ) by a clever use of data structures in the final step; for ease of exposition, we omit the details.
Finally, since Algorithm 2 is applied to each data point i ∈
[ N ] , the overall complexity increases by a factor of N .
We provide additional intuition behind Algorithm 2 immediately before the proof of Theorem 5 in Appendix E.
Numerical Results
Section 4.1 first compares the runtimes of our column-and-constraint scheme from Section 3 with those of solving the DR logistic regression problem (4) naïvely as a monolithic exponential conic program.
We subsequently compare the classification performance of problem (1) with those of a classical unregularized and regularized logistic regression on standard benchmark instances with categorical features (Section 4.2) and mixed features (Appendix C).
All algorithms were implemented in Julia [5] (MIT license) and executed on Intel Xeon 2.66GHz processors with 8GB memory in single-core mode.
We use MOSEK 9.3 [26] (commercial) to solve all exponential conic programs through JuMP [12] (MPL2 License).
(We note the open source solvers
Ipopt and CVXOpt could be used instead of MOSEK.)
All source codes and detailed results are available on GitHub ( https://github.com/selvi-aras/WassersteinLR ).
4.1 Runtime Comparison with Monolithic Formulation
We first compare the computation times of our column-and-constraint scheme from Section 3 with those of solving the DR logistic regression problem (4) naïvely as a monolithic exponential conic program.
To this end, we randomly generate synthetic logistic regression instances with varying numbers N of data points and m of binary features.
While Figure 2 (left) shows that both approaches scale similarly in the number N of data points, Figure 2 (middle) reveals that the solution of the monolithic formulation scales exponentially in the number m of binary features.
In contrast, our column-and-constraint scheme scales gracefully in both the number N of data points and the number m of binary features ( cf.
Figure 2, right).
A similar behavior can be observed with more general, non-binary categorical features; we omit the results due to space constraints.
4.2 Performance on Categorical-Feature Instances
We next compare the classification performance (in terms of the out-of-sample classification error) of our unregularized (‘DRO’) and Lasso-regularized (‘r-DRO’) DR logistic regression problem (1) with those of a classical unregularized (‘LR’) as well as Lasso-regularized (‘r-LR’), mass transportationregularized (MT) [31, 32], and robust Wasserstein profile inference-regularized (PI) [7] logistic
[Table]
regression on the 14 most popular UCI data sets that only contain categorical features having more than 30 rows [11] (varying licenses).
Instances with multiple output labels are indicated with a star; we convert them into instances with binary output labels by distinguishing between the majority class vs. all other classes.
The instances vary in the number N of data points, the number m of categorical features as well as, accordingly, the number k of slopes considered in the one-hot encoding of the categorical features ( cf.
Section 2.1).
All results are reported as means over 100 random training set-test set splits (80%:20%).
The radius ϵ ∈ { 0 , 10 − 5 , . . .
, 10 − 4 , . .
, 1 } of the Wasserstein ball as well as the Lasso penalty γ ∈ { 0 , 12 · 10 − 5 , . .
, 12 · 10 − 4 , . .
. , 12 } are selected via 5-fold crossvalidation.
We consider two variants of our DR logistic regression that employ a different output label weight ( κ = 1 vs. κ = m ) in the ground metric ( cf.
Definition 2).
The results are reported in Table 1. (Run-times and are provided in Appendix D where we also describe how statistical significance was assessed.)
The table shows that for the unregularized model, the classical logistic regression achieves the lowest classification error in 21% of the instances, whereas our DR logistic regression achieve the lowest classification error in 36% and 79% of the instances for κ = 1 , m , respectively.
(The double-counting of ties means these percentages do not sum to 100%.)
For the regularized model, the results change to 14% (classical logistic regression) vs. 57% (each of our models).
Within the numerical-feature Wasserstein DRO benchmarks, PI achieves the lowest classification error in 14% of the instances, whereas MT achieves in 42% and 57% of the instances for κ = 1 , m , respectively.
Globally, one of our methods is the winning approach in 12 out of 14 datasets (in the remainingthey are the second best approach without statistically significant inferiority).
In 9 of the datasets our methods win strictly (without a tie), and in 7 of these datasets the improvements are statistically significant over all other approaches.
Conclusions
We proposed a new DR mixed-feature logistic regression model where the proximity between the empirical distribution and the unknown true data-generating distribution is measured by the popular Wasserstein distance.
Despite its exponential-size formulation, we prove that the underlying optimization problem can be solved in polynomial time, and we develop a practically efficient column-and-constraint generation scheme for its solution.
The promising performance of our model is demonstrated in numerical experiments on standard benchmark instances.
We note that our columnand-constraint scheme readily extends to other DR mixed-feature machine learning models such as linear regression, support vector machines and decision trees.
Applying our algorithm in those settings is a promising direction for future research.
A further interesting direction is the development of tailored solution schemes rather than using off-the-shelf solvers for solving our DR mixed-feature logistic regression problem.
Acknowledgments and Disclosure of Funding
Funding from the EPSRC grant EP/W003317/1 is gratefully acknowledged.
The first author acknowledges support from The Alan Turing Institute.
The authors are grateful for the comments and suggestions of the anonymous reviewers, which have helped to significantly improve the manuscript.
This work has been produced without the involvement of any commercial entities that may cause a conflict of interest or competing interests.