Trimmed Maximum Likelihood Estimation for Robust Learning in Generalized Linear Models
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We study the problem of learning generalized linear models under adversarialcorruptions.
We analyze a classical heuristic called the iterative trimmed maximumlikelihood estimator which is known to be effective against label corruptions inpractice.
Under label corruptions, we prove that this simple estimator achievesminimax near-optimal risk on a wide range of generalized linear models, including
Gaussian regression, Poisson regression and Binomial regression.
Finally, weextend the estimator to the much more challenging setting of label and covariatecorruptions and demonstrate its robustness and optimality in that setting as well.
Introduction
Generalized linear models (GLMs) are an elegant framework for statistical modeling of data and arewidely used in many applications [23, 13, 21].
A generalized linear model captures the relationshipof labels (or observations)
y to covariates x via the conditional density f
( y | β (cid:62) x ) ∝ exp( y · β (cid:62) x −b ( β (cid:62) x )) .
Here β is the parameter of the model.
Parameter estimation in GLMs is done via thestandard maximum likelihood estimation (MLE) paradigm and has been extensively studied [23].
While GLMs offer a mathematically tractable formulation for statistical modeling, real data rarelysatisﬁes the generative process of a GLM and as a result there has been considerable interest indeveloping robust learning algorithms for GLMs [20, 22, 25].
The predominant way to modelmisspeciﬁcation or adversarial corruptions in the data is Huber’s (cid:15) -contamination model [16] andits recent extensions that allow for stronger adversaries [11].
These models assume that a small (cid:15)fraction of the data is corrupted by an adversary.
Furthermore, the adversary can be restricted toeither only corrupting the labels y , or be allowed to corrupt both the covariates x and the labels y foran (cid:15) fraction of the data.
Various algorithms have been proposed for robust estimation in generalized linear models.
One lineof work proposed computationally intractable algorithms that are based on non-convex M estimators[16, 20] or by running tournaments over an exponentially large search space [29].
Another line ofwork proposes polynomial time algorithms that either achieve sub-optimal error rates [25] or onlyapply to restricted settings such as the noise being heavy tailed [31].
In practical settings a simpleheuristic namely the iterative trimmed estimator has been shown to work well under settings whereonly the labels are corrupted [26].
However, from a theoretical perspective the iterative trimmed
MLE estimator has only been analyzed under restrictive settings such as when the underlying GLMis a Gaussian regression model.
Our key theoretical contribution in this work is a general analysis of the trimmed MLE estimator.
In particular, we show that for a broad family of GLMs, and under adversarial corruptions of onlythe labels, not only does the iterative trimmed MLE estimator enjoy theoretical guarantees, it in factachieve the minimax error rate!
Next, we also consider the more challenging setting of corruptions to this case, the recent work of
], we modify the iterative trimmed MLE estimator to propose a polynomial time algorithm based on semi-deﬁnite programmingthat can simultaneously handle both covariate and label corruptions and nearly achieve minimax errorrates.
Below we state our main results.
Theorem 1.1 (Informal Theorem) .
Let S (cid:15) =
{ ( x 1 , y 1 ) , . . .
, ( x n , y n ) } be independent and identicallydistributed samples generated by a generalized linear model with sub-Gaussian x i .
Let an (cid:15) fractionof the labels be adversarially corrupted.
Then, with high probability, the iterative trimmed MLEestimator when given as input S (cid:15) provides the following guarantees:
• O ( σ(cid:15) log(1 /(cid:15) ))
parameter estimation error for the Gaussian regression model where σ 2 isthe variance of Gaussian noise on y .•
O ( (cid:15) exp( (cid:112) log(1 /(cid:15) )))
parameter estimation error for Poisson regression model. • O ( 1 √ m (cid:15) (cid:112) log( m/ε ) log(1 /ε ))
parameter estimation error for the Binomial regressionmodel with m trials. • O ( (cid:15) log(1 /(cid:15) ))
parameter estimation error for a general class of smooth and continuousGLMs (includes the Gaussian regression model).
Previous work [26] analyze the iterative trimmed maximum likelihood estimator under the Gaussianregression settings (least square) where only the labels are corrupted, and proved a O ( σ ) (cid:96) 2 -errorbound for parameter estimation.
Other iterative trimming approach such as [4] also achieves onlyachieve O ( σ ) error in this setting.
Our error bound of O ( σε log(1 /ε )) signiﬁcantly improved thedependency on the corruption level ε and nearly matches the minimax lower bound of σε [15].
Inall the generalized linear models studied in this work, we show that the iterative trimmed maximumlikelihood estimator achieves O ( ε 1 − δ ) error for any δ > 0 , which matches the minimax lower boundΩ( ε ) up to a sub-polynomial factor.
Next, we present our second main result that can simultaneously handle both covariate and labelcorruptions and nearly achieve minimax error rate.
Theorem 1.2 (Informal Theorem) .
Let S (cid:15) =
{ ( x 1 , y 1 ) , . . .
, ( x n , y n ) } be independent and identicallydistributed samples generated by a generalized linear model with sub-Gaussian x i .
Let an (cid:15) fractionof the labels and covariates be adversarially corrupted.
Then there is a polynomial time algorithmbased on semi-deﬁnite programming that takes S (cid:15) as input and with high probability achieves thesame parameter estimation recovery bounds as in Theorem 1.1 above.
Our algorithm requires the covariance matrix of the covariate distribution being identity or known.
Thus the error rate we get is incomparable to the results in the general covariance settings.
Outline of the paper.
In Section 2 we discuss related work.
We deﬁne preliminaries in Section 3followed by the iterative trimmed MLE algorithm, formal results, and proof sketches for the labelcorruption case in Section 4.
In Section B, we introduce our algorithm, results and proof sketches forthe sample corruption case.
We defer proofs to the Appendix.
Related Work
There is a large body of work both on the theoretical and practical aspects of generalized linearmodels. Here we discuss the works most relevant to the current setting that we study. Classical worksin the Huber’s contamination model present algorithms for general robust estimation that obtainnear optimal error rates [16, 29]. However these algorithms are computationally inefﬁcient as theyrequires enumeration over an exponentially large search space.
Polynomial time algorithms for robustestimation in GLMs have also been proposed. The recent work of [31] propsoed a reweighted MLEestimator for dealing with settings where the covariates are heavy tailed. The work of [25] proposesa general procedure for robust gradient descent and shows that one can use this to design robustestimation algorithms for GLMs. However the bounds obtained have sub-optimal dependence on theerror rate (cid:15) and also have a mild dimension dependence.
In particular, the resulting algorithms cannothandle the case of constant corruption rates. In another line of work, a simple heuristic namely thetrimmed MLE estimator has been shown to perform well under corruptions to observation [26]. Therealso exist theoretical guarantees for this heuristic under the Gaussian regression model in which case MLE reduces to standard linear least squares regression problem [26].
There is also a vast amount of literature in statistics, machine learning and theoretical computerscience on algorithms that are robust to covariate corruptions. Minimax optimal but computationallyinefﬁcient robust estimators have been established in the works of [28, 29, 6, 15] for a variety ofproblems such as mean and covariance estimation. In recent years there has also been a line of workin designing computationally efﬁcient algorithms for handling adversarial corruptions [19, 11, 5, 4, 7].
There is also a long line of work on designing robust algorithms for the linear least squares problem.
This corresponds to the special case of the GLM being a Gaussian regression model [4, 3, 18, 8].
Another special case of GLMs that has been studied from a robustness perspective is the logisticregression model [14, 25].
Our general analysis of the iterative trimmed MLE estimator matchesthe best known error guarantees for this setting.
The recent work of [10] also studies learning of
GLMs under covariate corruptions.
Building upon the works of [25, 12] the authors present nearlylinear time algorithms that achieve an O ( √ (cid:15) ) error guarantee under bounded 4 th moment assumptionon the marginal of x .
On the other hand, our main contribution is to show that a simple heuristicachieves minimax optimal rates for a variety of GLMs under observation corruptions, and can beeasily modiﬁed to handle covariate corruptions as well.
Preliminaries
In this section, we formally introduce the robust generalized linear model studied in this paper.
Firstwe deﬁne the classical generalized linear model as follows.
Deﬁnition 3.1 (Generalized linear model) .
We say that ( x , y ) follows from a generalized linear model if there exists function b ( · ) , function c ( · ) and scalar φ such that the probability density function of y equals
[Equation]
The derivative b (cid:48) ( · ) is called mean function since E [ y | β (cid:62) x ] =
b (cid:48) ( β (cid:62) x ) .
The second order derivativeb (cid:48)(cid:48) ( · ) is called variance function since Var[ y | β (cid:62) x ] = b (cid:48)(cid:48) ( β (cid:62) x )
Here we provide three commonly used examples of generalized linear model studied in this paper.
Deﬁnition 3.2.
Commonly used examples of generalized liner model
• Gaussian regression : b ( θ ) = 12 θ 2 , c ( y ) =
exp( − y 2 / 2) √ 2 π
• Poisson regression : b ( θ ) =
exp( θ ) , c ( y ) =
1 y !
• Binomial regression : Let m be the number of trials of the binomial distribution.
b ( θ ) =
[Equation]
We consider sub-Gaussian random design setting in this work, i.e., each covariate x i is drawn i.i.d.from a sub-Gaussian distribution with zero mean and covariance Σ .
Deﬁnition 3.3 (Sub-Gaussian design) .
We assume that each covariate x i is drawn independently from a zero mean, covariance Σ sub-Gaussian distribution with sub-Guassian norm 1 , namely, E [ x i ] = 0 , E
[ x i x (cid:62) i ] = Σ and for all v ∈ R d
[Equation]
Having introduced the generation model of the good data, now we describe the corruption modelwhere the adversary is allowed to corrupt a small fraction of the data points.
Deﬁnition 3.4 (Corruption model) .
We consider two different data generation model with ε fractionof adversarial corruptions:
• Label corruption model :
Given n i.i.d.
sample { ( x i , y i ) } ni =1 generated by a generalized linear model.
The adversary is allowed to inspect the sample, and replace a total of ε · nlabels y i with arbitrary values.
• Sample corruption model:
Given n i.i.d.
sample { ( x i , y i ) } ni
=1 generated by a generalizedlinear model.
The adversary is allowed to inspect the sample, and replace a total of ε · ndata points ( x i , y i ) with arbitrary values.
We call the corrupted dataset S
=
T ∪ E where T contains the set of remaining uncorrupted datapoints, and E contains the set of data points that is controlled by the adversary.
The goal of our algorithm is recovering the underlying regression coefﬁcient β ∗ from a set ofcorrupted examples under (cid:96) 2 error metric, i.e., (cid:107) ˆ β − β ∗ (cid:107) .
We assume (cid:107) β ∗ (cid:107) ≤ R for a constant R .
For simplicity of the presentation, though out this paper, we assume Σ = I d .
We remark that ouralgorithm for label corruption model applies to the general covariance setting and is able to achievesmall estimation error in terms of (cid:107) Σ 1 / 2 (ˆ β − β ∗ ) (cid:107) 2 while the algorithm for sample corruption modelonly works with the knowledge of Σ .
Label Corruption
In this section, we formally describe our algorithm and proof-sketch for the label corruption setting.
4.1 Algorithm
We start with deﬁning the trimmed maximum likelihood estimator, that will be used in our algorithm.
Deﬁnition 4.1 (Trimmed maximum likelihood estimator) .
Given a set of data points S
= { ( x i , y i ) } ni =1 , deﬁne the trimmed maximum likelihood estimator as
[Equation]
In the setting of generalized linear model, the objective of trimmed maximum likelihood estimatoris a biconvex problem in ˆ S and β but not jointly convex.
The following alternating minimizationalgorithm is a simple heuristic to approximate the trimmed maximum likelihood estimator whosesimilar form has been studied in [26, 7]. Input: Set of examples S = { ( x 1 , y 1 ) , . . . , ( x n , y n ) } , ε , η , R
Output do Choose = arg min T ⊂ S (0) : | T | =(1 − 2 ε ) n (cid:80) i ∈ T − log f
( y i |(cid:104) ˆ β ( t ) , x i (cid:105) ) ;
Compute ˆ β
1) =
arg min β, (cid:107) β (cid:107)≤ R (cid:80) i ∈ ˆ S t ) − log f
( y i |(cid:104) β x
i (cid:105) )
; if 1 n i
S log
( y i
(cid:104)
ˆ β ( t +1)
x i (cid:105) ) > 1 n (cid:80) i ∈ ˆ S t ) − log f
( y i |(cid:104) ˆ β ( t ) , x i (cid:105) )
− η then β
The algorithm starts by naively pruning out ε · n data points with the largest labels.
Then each roundof the alternating minimization algorithm has two steps.
In optimizing over S , we ﬁnd the set ˆ S ( t )of size (1 − ε ) n with the best likelihood.
In optimizing over β , we ﬁnd regression coefﬁcient β ( t )which maximize the likelihood on the current set of data ˆ S ( t ) .
The algorithm terminate and output βwhen the likelihood no longer improves by more than η .
It is clear that the algorithm terminate in
O (1 /η ) rounds when the log-likelihood is bounded.
Since the original trimmed maximum likelihoodestimator is a biconvex optimization problem in S, β which is not jointly convex, our algorithm doesnot guarantee to return a global optimal solution.
Nonetheless, as we showed, it does return a ﬁrstorder stationary point which will be close to the true coefﬁcient β ∗
Theorem 4.2 (Gaussian regression with label corruption) .
Let S
= { x i , y i } ni =1 be a set of data points generated by a Gaussian regression model with sub-Gaussian design, with ε c -fraction of label corruption and n = Ω( d +log(1 /δ )
ε) .
With probability 1 − δ , Algorithm 1 with parameters ε = ε c , η = ε 2 c , R = ∞ terminate within O (1 /ε 2 c ) iterations, and output an estimate ˆ β such that
[Equation]
Theorem 4.3 (Poisson regression with label corruption) .
Let S
= { x i , y i } ni =1 be a set of data points generated by a Poisson regression model with sub-Gaussian design, with ε c -fraction of label corruption and n = Ω( dε 2 ) .
With probability 0 .
99 , Algorithm 1 with parameters ε =
ε c , η = ε 2 c / ( dn ) , R ≥ (cid:107) β ∗ (cid:107) terminate within dn/ε 2 c iterations, and output an estimate ˆ β such that
[Equation]
Theorem 4.4 (Binomial regression with label corruption) .
Let S
= { x i , y i } ni =1 be generated by a Binomial regression model with Sub-Gaussian Design, with ε c -fraction of label corruption and n = Ω( d +log(1 /δ )
ε) . With probability 1 − δ , Algorithm 1 with parameters ε = ε c , η = ε 2 c , R ≥ (cid:107) β ∗ (cid:107) terminate within m/ε 2 c iterations, and output an estimate ˆ β such that
[Equation]
4.2 Proof Sketch
We provide an intuitive proof sketch for the above theorems (the full proofs are deferred to Section
A in the Appendix)
The guarantee of our alternating minimization algorithm relies on two claims: First, the algorithmreturns an approximate stationary point
ˆ β .
Second, any approximate stationary point will be close tothe true coefﬁcient β ∗ .
In this section, we will present high level intuition of the proof of the twoclaims.
Alternating minimization algorithm returns an approximate stationary point.
First we deﬁneﬁrst order approximate stationary point as follows.
Let ˆ β ∈ R
d be a regression coefﬁcient vector andˆ S contains the set of datapoints of size (1 − ε ) n with the largest log-likelihood under ˆ β .
We call ˆ β aγ -approximate stationary point if
[Equation]
i.e., the gradient of the log-likelihood projected along the β ∗ − ˆ β direction is small.
Our goal is toshow when the algorithm terminate, that is when ˆ β can not be improved by more than η , the gradientalong the β ∗ − ˆ β must be small.
This is clear where the empirical log-likelihood function is smooth,simply because if the gradient is large, one can improve the log-likelihood by more than η which willresult in a contradiction.
The smoothness (norm of the Hessian matrix) of the empirical log-likelihoodin generalized linear model is directly related to the range of b (cid:48)(cid:48)
( θ ) .
In particular, b (cid:48)(cid:48) ( θ ) is boundedfor Gaussian regression and Binomial regression.
However, the problem arise for Poisson regression where b (cid:48)(cid:48)
( θ ) =
exp( θ ) becomes extremely largefor large θ , which results in non-smooth curvature for empirical log-likelihood.
We overcome thisdifﬁculty by leveraging the special property of function b ( θ ) =
exp( θ ) in the Poisson regressionsetting.
Observing that the derivative b (cid:48) ( θ ) is equal to second order derivative b (cid:48)(cid:48)
( θ ) for Poissonregression, the gradient along the β ∗ − ˆ β direction can not be small when the second order derivativealong the β ∗ − ˆ β direction gets large, which will result in a more than η improvement of the loglikelihood by moving toward β ∗ and a contradiction.
Therefore, the second order derivative along theβ ∗ − ˆ β direction is small, and we are back to the smooth objective function setting.
Any approximate stationary point will be close to the true coefﬁcient.
Let us ﬁrst write down theγ -approximate stationary condition for generalized linear model as n (cid:88)
[Equation]
Recall that T contains the set of uncorrupted data points, and E contains the set of data points that iscontrolled by the adversary.
Split ˆ S into ˆ S ∩ T and ˆ S ∩ E , and rearrange the terms we get n
[Equation]
To obtain an upper bound on (cid:107) β ∗ − ˆ β (cid:107) , we will prove a lower bound in terms of (cid:107) β ∗ − ˆ β (cid:107) on the lefthand side, and an upper bound terms of (cid:107) β ∗ − ˆ β (cid:107) on the right hand side.
Finally we will combine theupper and lower bound into an upper bound on (cid:107) β ∗ − ˆ β (cid:107) .
Lower bound on the LHS.
Note that ˆ S ∩ T contains uncorrupted data points.
The high levelintuition is that since mean of y i is b (cid:48) ( β ∗(cid:62) x i ) and b (cid:48) ( · ) is monotone, (cid:16)
y
i − b (cid:48) ( β ∗(cid:62) x i ) (cid:17) ( β ∗ − ˆ β ) (cid:62) x ishould be roughly (cid:16) ( β ∗ − ˆ β ) (cid:62) x
i (cid:17) 2 , and (cid:16) ( β ∗ − ˆ β ) (cid:62) x i (cid:17) 2 should be proportional to (cid:107) β ∗ − ˆ β (cid:107) 2given enough samples.
More formally, we will decompose the LHS as
[Equation]
The ﬁrst term contains a (1 − ε ) fraction of uncorrupted examples sampled from a zero meandistribution with certain tail bound, e.g. sub-exponential for Gaussian and Binomial regression, k -thmomment bound for Poisson regression.
Therefore, we can apply resilience property to bound the ﬁrst term.
Over this work, we heavily utilizethe resilience property of the sample set that is drawn from “nice” distributions.
Take sample meanas an example, resilience [27, 30] dictates that given a large enough sample set S
= { x i } ni =1 , thesample mean of any large enough subset of S will be close to each other.
Speciﬁcally, under subGaussian distribution, for any T ⊂ S, | T | ≥ (1 − ε ) n , it holds that (cid:107) 1 | T | (cid:80) i ∈ T x
i − 1 | S | (cid:80) i ∈
S x i (cid:107) ≤ε (cid:112) log(1 /ε ) .
Resilience property applies to sub-exponential and k -th moment bounded distributionas well, and this gives us a way to control the behavior of any subset of good data.
For the second term, we prove that 1 n (cid:80) i ∈ ˆ S ∩ T b ( β (cid:62) x i ) is a strongly convex function again using theresilience property, which implies 1 n (cid:80) i ∈
ˆ S ∩ T (cid:16) b (cid:48) ( β ∗(cid:62) x i ) − b (cid:48) (ˆ β (cid:62) x i ) (cid:17) ( β ∗ − ˆ β ) (cid:62) x i = Ω( (cid:107) β ∗ −ˆ β (cid:107) 2 )
Upper bound on the RHS.
To upper bound − 1 n (cid:80) i ∈ ˆ S ∩ E ( y i − b (cid:48)
(ˆ β (cid:62) x i ))
( β ∗ − ˆ β ) (cid:62) x i , we will prove an upper bound on (cid:113) 1 n (cid:80) i ∈ ˆ S ∩ E
( y i − b (cid:48)
(ˆ β (cid:62) x i ))
2 and (cid:113) 1 n (cid:80) i ∈ ˆ S ∩ E (( β ∗ − ˆ β ) (cid:62) x i ) 2 , then apply Cauchy-Schwarz inequality.
The key difﬁculty is bounding (cid:113) 1 n (cid:80) i ∈ ˆ S ∩ E
( y i − b (cid:48) (ˆ β (cid:62) x i )) 2 , as it contains corrupted data points controlled by an adversary, which does not follow any good property possessed by the good stochastic data.
However, since ˆ S contains (1 − ε ) n datapoints with the largest log-likelihood under ˆ β , we can argue that
[Equation]
since otherwise one can replace the data point in ˆ S ∩ E by the ones in T \ ˆ S to form a new setwith better likelihood than ˆ S .
This gives us an upper bound on the negative log-likelihood of y i inˆ S ∩ E .
Therefore we adopt a two step approach to upper bound the (cid:113) 1 n (cid:80) i ∈ ˆ S ∩ E (( β ∗ − ˆ β ) (cid:62) x i ) 2 .
First we prove an upper bound on the negative log-likelihood on T \ ˆ S , which becomes a negativelog-likelihood bound on ˆ S ∩ E immediately.
Second we turn the negative log-likelihood bound introa square error bound.
The two steps varies drastically for different regression models.
For the ﬁrst step of upper boundingthe negative log-likelihood, in Gaussian regression we use the resilience property of the quadratic form of sub-Gaussian random variable.
In Poisson regression, we leverage the resilience propertyof y i x i .
In Binomial regression, since the distribution only has support size m , we directly analyzethe resilience of the negative log-likelihood.
For the second step of turning the log-likelihood boundto a quadratic bound, we get the bound trivially in Gaussian regression since Gaussian likelihood isindeed quadratic.
For Poisson and Binomial setting, we have to build a proxy function which lowerbound the negative log-likelihood function − log f
( y i |(cid:104) ˆ β, x i (cid:105) ) to connect it to quadratic function.
4.3 Proof for Poisson Regression
As an illustrative example, we show how the above proof sketch can be used for Poisson regression toformally prove Theorem 4.3.
(Proofs for the other GLMs are deferred to Section A in the Appendix)
Lemma 4.5 (Approximate stationary point close to β ∗ for Poisson regression) .
Given as set of datapoints S = { x i , y i } ni =1 generated by a Poisson regression model with ε -fraction of label corruption, and the largest εn labels removed.
Let ˆ β be a max( ε, ε 2 / (cid:107) β ∗ − ˆ β (cid:107) )
-stationary point and (cid:107) ˆ β (cid:107) ≤ R .
Given that n = Ω( dε 2 ) , with probability 0 . 99 , it holds that
[Equation]
Since b (cid:48)(cid:48)
( θ ) =
exp( θ ) is unbounded for Poisson regression, the following lemma (proved in theappendix) shows that alternating minimization algorithm still return an approximate stationary point
Lemma 4.6 (Algorithm 1 ﬁnds an approximate stationary point for Poisson regression) .
Given asset of datapoints S
= { x i , y i } ni =1 generated by a Poisson model with ε c -fraction of corruption.
Assuming that n = Ω( d +log(1 /δ )
ε) , then with probability 1 − δ , the output of Algorithm 1 with inputparameters ε =
ε c , R ≥ (cid:107) β ∗ (cid:107) , η = ε 2 / ( dn ) , is a max( ε, 2 ε 2 (cid:107) β ∗ − ˆ β (cid:107) )
-approximate stationary point.
Proof of Theorem 4.3.
Lemma 4.6 implies the output of Algorithm 1 is a max( ε, ε 2 (cid:107) β ∗ − ˆ β (cid:107) )
approximate stationary point.
Lemma 4.5 then implies that (cid:107) ˆ β − β ∗ (cid:107) =
O ( ε exp( (cid:112) log(1 /ε ))) .
To bound thenumber of iterations, we need an upper bound on the negative log-likelihood on β = 0 , and a uniformlower bound on the negative log-likelihood.
The initial negative log-likelihood is upper bounded by1 n (cid:80)
i ∈
ˆ S (1) log( y i !)
+ 1 ≤ O ( E [ y 2 i ] + 1) =
O (1) where S (1) contains the smallest (1 − ε ) n labels.
Trivially, there is a 0 lower bound on the negative log-likelihood for Poisson distribution.
Therefore,the algorithm will terminate in dn/ε 2 c iterations.
Result for Sample Corruption Model
The learning problem becomes much harder in the presence of label and covariate corruption, since itbecomes much harder to tell whether a data point is corrupted by simply looking at the likelihood ofy i .
From a technical level, the resilience condition we leveraged on covariate x
i in set E breaks downwhen there is covariate corruption.
Luckily, we are able to restore the resilience property by addinga positive semideﬁnite (PSD) constraint when choosing the set ˆ S ( t ) .
Speciﬁcally, if the covariatedistribution has identity (or known) covariance and sub-Gaussian tail, one can add the PSD constraint n (cid:80) w
i x i x (cid:62) i (cid:22) (1 + ε log(1 /ε ))
· I to the ε corrupted data set, and the resulting weighted data set{ w i x i } ni =1 will have the same resilient condition as an uncorrupted data set.
Similar approaches havebeen applied be applied before in robust mean estimation [9] and robust regression [7].
To adapt our alternating minimization algorithm to the sample corruption model, we replace the stepof choosing the best ﬁt subset ˆ S ( t ) by a SDP program, which intuitively tries to ﬁnd the set with thebest likelihood and has bounded covariance at the same time.
This is a covering/packing SDPs whichcan be solved in nearly linear time [24, 2].
5.1 Algorithm
The guarantee of Algorithm 2 is formalized in the following theorems.
Our approach can be extendsto other generalized linear models, as we showed in Appendix B.
Input: Set of examples S
= { ( x 1 , y 1 ) , . . .
, ( x n , y n ) } , ε, η, R
Output: ˆ β arg do do
Let ˆ w ( t ) be the solution of the following SDP arg min w n (cid:88) i = − w i log f
{+( y i |(cid:104) ˆ β ( t ) , x i (cid:105) ) s.t. 0 ≤ w i ≤ 1 ∀ i ∈ S (0) w i = 
 ∀ i (cid:54)∈ S (0) (cid:107) w (cid:107) 
+}
= (1 − 2 ε ) n n (cid:88) i ∈ [ n ] w i x i x (cid:62) i (cid:22) (1 + ε log(1 /ε )) · I
Compute ˆ β ( t +1) =
arg min β, (cid:107) β (cid:107)≤ R (cid:80) ni
=1 − ˆ w ( t ) i log f
( y i |(cid:104) β, x i (cid:105) )
; if (cid:80) ni =1 − ˆ w ( t ) i log f ( y i |(cid:104) ˆ β ( t +1) , x i (cid:105) ) > (cid:80) ni
=1 − ˆ w ( t ) i log f
( y i |(cid:104) ˆ β ( t ) , x i (cid:105) )
− η then
Return β ( t )
Theorem 5.1 (Gaussian regression with sample corruption) .
Given as set of datapoints S = { x i , y i } ni =1 generated by a Gaussian regression model with sub-Gaussian design, ε c -fraction of sample corruption and n = Ω( d +log(1 /δ )
[Equation]
With probability 1 − δ , Algorithm 2 with parameters ε = ε c , η = ε 2 c , R = ∞ terminate within O (1 /ε 2 c ) iterations, and output an estimate ˆ β such that
[Equation]
Theorem 5.2 (Binomial regression with sample corruption) .
Given as set of datapoints S = { x i , y i } ni =1 generated by a Binomial regression model with ε c -fraction of label corruption and n = Ω( d +log(1 /δ ) ε) .
With probability 1 − δ , Algorithm 2 with parameters ε = ε c , η = ε 2 c , R ≥ (cid:107) β ∗ (cid:107) terminate within m/ε 2 c iterations, and output an estimate ˆ β such that
[Equation]
5.2 Proof Sketch
The proof is very similar to the label corruption setting, except that instead of applying resiliencecondition to subset of T , we need the same condition on the weighted subset of T .
For examplereplacing 1 | [Equation] i [Equation]
ˆ
[Equation]
x i by
[Equation]
i ∈ T
w i x i .
Luckily, all the resilience condition applies tothe weighted subset as well.
Compared to the label corruption setting, speciﬁcally there are two places we need to change in theargument.1.
Since x i , i ∈ E is now controlled by the adversary, we can no longer bound(cid:113)
1 n (cid:80) i ∈ ˆ S ∩ E (( β ∗ − ˆ β ) (cid:62) x i ) 2 by the resilience property of sub-Gaussian samples.
Instead,we leverage the fact that corrupted sample with small covariance is also resilient.2.
Previously, T is trivially a feasible solution in the step of choosing ˆ S .
Now we need to argue
Conclusion
In this paper we provided a general theoretical analysis showing that a simple and practical heuristicnamely the iterative trimmed MLE estimator achieves minimax optimal error rates under labelcorruptions for a wide class of generalized linear models (GLMs).
We also designed algorithmsbased on semi-deﬁnite programming (SDP) to make the estimator robust to covariate corruptionsas well.
While our SDP based algorithm runs in polynomial time, it would be interesting to exploremore practical variants that can handle covariate corruptions.
It would also be interesting to studywhether our techniques can be extended to design robust algorithms for more general exponentialfamilies beyond GLMs.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (See Section 6)
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[N/A]
(b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
[N/A]
(c) Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]