Trimmed Maximum Likelihood Estimation for Robust Learning in Generalized Linear Models
Anonymous Author(s)
AfﬁliationAddress
Abstract
We study the problem of learning generalized linear models under adversarialcorruptions.
We analyze a classical heuristic called the iterative trimmed maximumlikelihood estimator which is known to be effective against label corruptions inpractice.
Under label corruptions, we prove that this simple estimator achievesminimax near-optimal risk on a wide range of generalized linear models, including
Gaussian regression, Poisson regression and Binomial regression.
Finally, weextend the estimator to the much more challenging setting of label and covariatecorruptions and demonstrate its robustness and optimality in that setting as well.
Introduction
Generalized linear models (GLMs) are an elegant framework for statistical modeling of data and arewidely used in many applications [33, 20, 31].
A generalized linear model captures the relationshipof labels (or observations)
y to covariates x via the conditional density f
( y | β (cid:62) x ) ∝ exp( y · β (cid:62) x −b ( β (cid:62) x )) .
Here β is the parameter of the model.
Parameter estimation in GLMs is done via thestandard maximum likelihood estimation (MLE) paradigm and has been extensively studied [33].
While GLMs offer a mathematically tractable formulation for statistical modeling, real data rarelysatisﬁes the generative process of a GLM and as a result there has been considerable interest indeveloping robust learning algorithms for GLMs [30, 32, 36].
The predominant way to modelmisspeciﬁcation or adversarial corruptions in the data is Huber’s (cid:15) -contamination model [24] andits recent extensions that allow for stronger adversaries [15].
These models assume that a small (cid:15)fraction of the data is corrupted by an adversary.
Furthermore, the adversary can be restricted toeither only corrupting the labels y , or be allowed to corrupt both the covariates x and the labels y foran (cid:15) fraction of the data.
Various algorithms have been proposed for robust estimation in generalized linear models.
One lineof work proposed computationally intractable algorithms that are based on non-convex M estimators[24, 30] or by running tournaments over an exponentially large search space [42].
Another line ofwork proposes polynomial time algorithms that either achieve sub-optimal error rates [36] or onlyapply to restricted settings such as the noise being heavy tailed [45].
In practical settings a simpleheuristic namely the iterative trimmed estimator has been shown to work well under settings whereonly the labels are corrupted [38].
However, from a theoretical perspective the iterative trimmed
MLE estimator has only been analyzed under restrictive settings such as when the underlying GLMis a Gaussian regression model.
Our key theoretical contribution in this work is a general analysis of the trimmed MLE estimator.
In particular, we show that for a broad family of GLMs, and under adversarial corruptions of onlythe labels, not only does the iterative trimmed MLE estimator enjoy theoretical guarantees, it infact nearly achieves the minimax error rate!
Next, we also consider the more challenging setting ofcorruptions to both covariates and labels.
In this case, building upon the recent work of [9], we modify the iterative trimmed MLE estimator to propose a polynomial time algorithm based on semi-deﬁniteprogramming that can simultaneously handle both covariate and label corruptions and nearly achieveminimax error rates.
Below we state our main results.
Theorem 1.1 (Informal Theorem) .
Let S (cid:15) =
{ ( x 1 , y 1 ) , . . .
, ( x n , y n ) } be independent and identicallydistributed samples generated by a generalized linear model with sub-Gaussian x i .
Let an (cid:15) fractionof the labels be adversarially corrupted.
Then, with high probability, the iterative trimmed MLEestimator when given as input S (cid:15) provides the following guarantees:
• O ( σ(cid:15) log(1 /(cid:15) ))
parameter estimation error for the Gaussian regression model where σ 2 isthe variance of Gaussian noise on y .•
O ( (cid:15) exp( (cid:112) log(1 /(cid:15) )))
parameter estimation error for Poisson regression model.• O ( 1 √ m (cid:15) (cid:112) log( m/ε ) log(1 /ε ))
parameter estimation error for the Binomial regressionmodel with m trials.• O ( (cid:15) log(1 /(cid:15) ))
parameter estimation error for a general class of smooth and continuousGLMs (includes the Gaussian regression model).
Previous work [38, 5] analyze the iterative trimmed maximum likelihood estimator under the Gaussianregression settings (least square) where only the labels are corrupted, and proved an O ( σ ) (cid:96) 2 -errorbound for parameter estimation.
Our error bound of O ( σε log(1 /ε )) signiﬁcantly improved thedependency on the corruption level ε and nearly matched the minimax lower bound of σε [23].
Inall the generalized linear models studied in this work, we show that the iterative trimmed maximumlikelihood estimator achieves O ( ε 1 − δ ) error for any δ > 0 , which matches the minimax lower boundΩ( ε ) up to a sub-polynomial factor.
Next, we present our second main result that can simultaneously handle both covariate and labelcorruptions and nearly achieve minimax error rate.
Theorem 1.2 (Informal Theorem) .
Let S (cid:15) =
{ ( x 1 , y 1 ) , . . .
, ( x n , y n ) } be independent and identicallydistributed samples generated by a generalized linear model with sub-Gaussian x i .
Let an (cid:15) fractionof the labels and covariates be adversarially corrupted.
Then there is a polynomial time algorithmbased on semi-deﬁnite programming that takes S (cid:15) as input and with high probability achieves thesame parameter estimation recovery bounds as in Theorem 1.1 above.
Our algorithm requires the covariance matrix of the covariate distribution being identity or known.
Thus the error rate we get is incomparable to the results in the general covariance settings.
Outline of the paper.
In Section 2 we discuss related work.
We deﬁne preliminaries in Section 3followed by the iterative trimmed MLE algorithm, formal results, and proof sketches for the labelcorruption case in Section 4.
In Section 5, we introduce our algorithm, results and proof sketches forthe sample corruption case.
We defer proofs to the Appendix.
Related Work
There is a vast amount of literature in statistics, machine learning and theoretical computer scienceon algorithms that are robust to adversarial corruptions and outliers.
Classical works in the Huber’scontamination model present algorithms for general robust estimation that obtain near optimalerror rates [24, 42].
Minimax optimal but computationally inefﬁcient robust estimators have beenestablished in the works of [41, 42, 7, 23] for a variety of problems such as mean and covarianceestimation.
In recent years there has also been a line of work in designing computationally efﬁcientalgorithms for handling adversarial corruptions [28, 15, 6, 5, 9, 14, 18].
Several special cases of generalized linear model has been studied extensively from the robustnessperspective.
There is a long line of work on designing robust algorithms for the linear least squaresproblem.
This corresponds to the special case of the GLM being a Gaussian regression model[40, 5, 4, 27, 10, 3, 35, 19].
When the covariate follows from a sub-Gaussian distribution, previousbest result [35] achieves (cid:96) 2 error σε (cid:112) log(1 /ε ) using Huber regression while we show iterativethresholding achieves σε log(1 /ε ) error.
Another special case of GLMs that has been studied froma robustness perspective is the logistic regression model [22, 36, 8].
Our analysis of the iterative trimmed MLE estimator on binomial regression matches the best known error guarantees for thissetting.
The work of [36] proposes a general procedure for robust gradient descent and shows that one can usethis to design robust estimation algorithms for a abroad class of GLMs.
Building upon these works,the authors in [25] present a nearly linear time algorithm for GLMs.
Compared to our work, [36]assumes x has bounded 8 th moment, [25] assumes x is 2 - 4 hypercontractive, and both papers achievean O ( √ (cid:15) ) error guarantee with ε -fraction of corruptions while our algorithm achieves a better O ( ε )guarantee under the stronger sub-Gaussian assumptions.
In addition, for GLMs, both papers assumea uniform upper bound (and lower bound) on the second order derivative of function b ( · ) , which isnot satisﬁed by the widely used Poisson and Binomial regression studied in this paper.
In a similarsetting, [45] proposed a reweighted MLE estimator for dealing with settings where the covariates areheavy tailed but not corrupted by an adversary.
Iterative thresholding is a longstanding heuristic for robust linear regression that dates back to
Legendre [29].
It’s theoretical property in the non-asymptotic regime is ﬁrst studied in [5], whichshows a O ( σ ) error bound for sufﬁciently small label corruption level when the label noise is N (0 , σ 2 ) .
The iterative thresholding algorithm is later extended and analyzed in the oblivious labelcorruption setting [4, 40], and is shown to provide consistent estimate even when the corruptionlevel goes to 1 . [35] extended the iterative thresholding algorithm to the heavy-tailed covariatesetting and can simultaneously handle both labels and covariate corruptions. In addition, [35] adaptedthe implicit result in [4] to show that iterative thresholding algorithm achieves O ( σ √ ε ) error ratefor sub-Gaussian covariate.
[38] empirically demonstrated the effectiveness of the trimmed lossestimator under labels corruptions, and also proved theoretical guarantees in a special class of GLMs,which, however, also has O ( σ ) error when specialized to the Gaussian noise setting.
[37] studied thetrimmed loss estimator in the mixed linear regression setting.
Finally, [9] proposed an alternatingminimization algorithm for the ﬁxed design linear regression setting with Huber contamination onthe labels, which is different from our strong contamination model (with adaptive replacement)for generalized linear model.
Their algorithm incorporates a semideﬁnite programming in the setselection step, and achieves a near-optimal ˜ O ( σε ) error.
On a very high level, our proof follows froma similar framework as in [9].[35] ﬁrst proposed a generic approach to modify an estimator which is robust against label corruptioninto one that is robust against simultaneous label and covariate corruptions by running a covariateﬁlter algorithm [17, 16] as a preprocessing step. In the Gaussian regression setting, our Algorithm 3is identical to Algorithm 3 of [35] and the difference is in the improved error rate. In particular, Lemma 4.1 in [35] implies a O ( σ √ ε ) error bound while we proved a O ( σε log(1 /ε )) error bound.
Preliminaries
In this section, we formally introduce the robust generalized linear model studied in this paper.
Firstwe deﬁne the classical generalized linear model as follows.
Deﬁnition 3.1 (Generalized linear model) .
We say that ( x , y ) follows from a generalized linear model if there exist function b ( · ) and function c ( · ) such that the probability density function of y equals
[Equation]
The derivative b (cid:48) ( · ) is called mean function since E [ y | β (cid:62) x ] =
b (cid:48) ( β (cid:62) x ) .
The second order derivativeb (cid:48)(cid:48) ( · ) is called variance function since Var[ y | β (cid:62) [Equation]
Here we provide three commonly used examples of generalized linear models studied in this paper.
Deﬁnition 3.2.
Commonly used examples of generalized linear model
• Gaussian regression : b ( θ ) = 1 2 θ 2 , c ( y ) =
exp( − y 2 / 2) √ 2 π•
Poisson regression : b ( θ ) =
exp( θ ) , c ( y ) =
1 y !
• Binomial regression : Let m be the number of trials of the binomial distribution.
b ( θ ) =
m log(1 + exp( θ )) , c ( y ) =
(cid:0) my (cid:1)
We consider the sub-Gaussian random design setting in this work, i.e., each covariate x i is drawni.i.d.
from a sub-Gaussian distribution with zero mean and covariance Σ .
Deﬁnition 3.3 (Sub-Gaussian design) .
We assume that each covariate x i is drawn independently from a zero mean, covariance Σ sub-Gaussian distribution with sub-Guassian norm 1 , namely, E [ x i ] = 0 , E
[ x i x (cid:62) i ] = Σ and for all v ∈ R d ,
Pr( v (cid:62) x i ≥ t ) ≤ exp( − t 2 ) .
Having introduced the generation model of the good data, now we describe the corruption modelwhere the adversary is allowed to corrupt a small fraction of the data points.
Deﬁnition 3.4 (Corruption model) .
We consider two different data generation model with ε fractionof adversarial corruptions:
• Label corruption model :
Given n i.i.d.
sample { ( x i , y i ) } ni
=1 generated by a generalizedlinear model.
The adversary is allowed to inspect the sample, and replace a total of ε · nlabels y i with arbitrary values.•
Sample corruption model:
Given n i.i.d.
sample { ( x i , y i ) } ni
=1 generated by a generalizedlinear model.
The adversary is allowed to inspect the sample, and replace a total of ε · ndata points ( x i , y i ) with arbitrary values.
We call the corrupted dataset S
=
T ∪ E where T contains the set of remaining uncorrupted datapoints, and E contains the set of data points that is controlled by the adversary.
The goal of our algorithm is recovering the underlying regression coefﬁcient β ∗ from a set of exampleswith ε fraction corrupted under (cid:96) 2 error metric, i.e., (cid:107) ˆ β − β ∗ (cid:107) .
We assume (cid:107) β ∗ (cid:107) ≤ R for a constant
R , and ε <
= c for a sufﬁciently small constant c .
For simplicity of the presentation, throughout thispaper, we assume Σ
= I d .
We remark that our algorithm for label corruption model applies to thegeneral covariance setting and is able to achieve small estimation error in terms of (cid:107) Σ 1 / 2 (ˆ β − β ∗ ) (cid:107) 2while the algorithm for sample corruption model only works with the knowledge of Σ .
Label Corruption
In this section, we formally describe our algorithm and proof-sketch for the label corruption setting.
4.1 Algorithm
We start with deﬁning the trimmed maximum likelihood estimator, which is a simple and naturalheuristic for robustly learning generalized linear model.
Deﬁnition 4.1 (Trimmed maximum likelihood estimator) .
Given a set of data points S
= { ( x i , y i ) } ni =1 , deﬁne the trimmed maximum likelihood estimator as
[Equation]
In the setting of generalized linear model, the objective of trimmed maximum likelihood estimatoris a biconvex problem in ˆ S and β but not jointly convex.
The following alternating minimizationalgorithm is a simple heuristic to approximate the trimmed maximum likelihood estimator whosesimilar form has been studied in [38, 9].
The algorithm starts by naively pruning out ε · n data points with the largest labels.
Then each roundof the alternating minimization algorithm has two steps.
In optimizing over S , we ﬁnd the set ˆ S ( t )of size (1 − ε ) n with the best likelihood.
In optimizing over β , we ﬁnd regression coefﬁcient β ( t )which maximizes the likelihood on the current set of data
ˆ S ( t ) .
The algorithm terminates and outputsβ when the likelihood no longer improves by more than η .
It is clear that the algorithm terminates in
O (1 /η ) rounds when the log-likelihood is bounded.
Since the original trimmed maximum likelihoodestimator is a biconvex optimization problem in S, β which is not jointly convex, our algorithm doesnot guarantee to return a global optimal solution.
Nonetheless, as we showed, it does return a ﬁrst order stationary point which will be close to the true coefﬁcient β ∗ .
It is worth noting that some recentpapers on robust statistics [13, 12, 43] show similar nice statistical properties of an approximate ﬁrstorder stationary point for non-convex optimization problems
We present the guarantee of our algorithm for Gaussian, Poisson and Binomial regression.
The resultfor other GLM is deferred to Appendix A.4.
Theorem 4.2 (Gaussian regression with label corruption) .
Let S
= { x i , y i } ni =1 be a set of data points generated by a Gaussian regression model with sub-Gaussian design, with ε c -fraction of label corruption and n = Ω( d +log(1 /δ )
ε) .
With probability 1 − δ , Algorithm 1 with parameters ε = ε c , η = ε 2 c , R = ∞ terminate within O (1 /ε 2 c ) iterations, and output an estimate ˆ β such that
[Equation]
Theorem 4.3 (Poisson regression with label corruption) .
Let S
= { x i , y i } ni =1 be a set of data points generated by a Poisson regression model with sub-Gaussian design, with ε c -fraction of label corruption and n = Ω( dε 2 ) .
With probability 0 .
99 , Algorithm 1 with parameters ε =
ε c , η = ε 2 c / ( dn ) , R ≥ (cid:107) β ∗ (cid:107) terminate within dn/ε 2 c iterations, and output an estimate ˆ β such that
[Equation]
Theorem 4.4 (Binomial regression with label corruption) .
Let S
= { x i , y i } ni =1 be generated by a
Binomial regression model with sub-Gaussian Design, with ε c -fraction of label corruption and n = Ω( d +log(1 /δ ) ε) .
With probability 1 − δ , Algorithm 1 with parameters ε = ε c , η = ε 2 c /m, R ≥ (cid:107) β ∗ (cid:107) terminate within m 2 /ε 2 c iterations, and output an estimate ˆ β such that
[Equation]
4.2 Proof Sketch
We provide an intuitive proof sketch for the above theorems (the full proofs are deferred to Section
A in the Appendix)
The guarantee of our alternating minimization algorithm relies on two claims: First, the algorithmreturns an approximate stationary point
ˆ β .
Second, any approximate stationary point will be close tothe true coefﬁcient β ∗ .
In this section, we will present high level intuition of the proof of the twoclaims.
Alternating minimization algorithm returns an approximate stationary point.
First we deﬁnethe ﬁrst order approximate stationary point as follows.
Let ˆ β ∈ R
d be a regression coefﬁcient vectorand ˆ S contains the set of datapoints of size (1 − ε ) n with the largest log-likelihood under ˆ β .
We callˆ β a γ -approximate stationary point if
[Equation]
i.e., the gradient of the log-likelihood projected along the β ∗ − ˆ β direction is small.
Our goal is toshow when the algorithm terminate, that is when ˆ β can not be improved by more than η , the gradientalong the β ∗ − ˆ β must be small.
This is clear where the empirical log-likelihood function is smooth,simply because if the gradient is large, one can improve the log-likelihood by more than η which willresult in a contradiction.
The smoothness (norm of the Hessian matrix) of the empirical log-likelihoodin generalized linear model is directly related to the range of b (cid:48)(cid:48)
( θ ) .
In particular, b (cid:48)(cid:48) ( θ ) is boundedfor Gaussian regression and Binomial regression.
However, a problem arises for Poisson regression where b (cid:48)(cid:48)
( θ ) =
exp( θ ) becomes extremely largefor large θ , which results in non-smooth curvature for empirical log-likelihood.
We overcome thisdifﬁculty by leveraging the special property of function b ( θ ) =
exp( θ ) in the Poisson regressionsetting.
Observing that the derivative b (cid:48) ( θ ) is equal to second order derivative b (cid:48)(cid:48)
( θ ) for Poissonregression, the gradient along the β ∗ − ˆ β direction can not be small when the second order derivativealong the β ∗ − ˆ β direction gets large, which will result in a more than η improvement of the loglikelihood by moving toward β ∗ and therefore a contradiction.
Hence, the second order derivativealong the β ∗ − ˆ β direction must be small, and we have the same argument as in the smooth objectivefunction setting.
Any approximate stationary point will be close to the true coefﬁcient.
Let us ﬁrst write down theγ -approximate stationary condition for generalized linear model as
[Equation]
Recall that T contains the set of uncorrupted data points, and E contains the set of data points that iscontrolled by the adversary.
Split ˆ S into ˆ S ∩ T and ˆ S ∩ E , and rearrange the terms we get n [Equation] i ∈ ˆ
[Equation]
To obtain an upper bound on (cid:107) β ∗ − ˆ β (cid:107) , we will prove a lower bound in terms of (cid:107) β ∗ − ˆ β (cid:107) on the lefthand side, and an upper bound in terms of (cid:107) β ∗ − ˆ β (cid:107) on the right hand side.
Finally we will combinethe upper and lower bound into an upper bound on (cid:107) β ∗ − ˆ β (cid:107) .
Lower bound on the LHS.
Note that ˆ S ∩ T contains uncorrupted data points.
The high levelintuition is that since mean of y i is b (cid:48) ( β ∗(cid:62) x i ) and b (cid:48) ( · ) is monotone, (cid:16)
y
i − b (cid:48) ( β ∗(cid:62) x i ) (cid:17) ( β ∗ − ˆ β ) (cid:62) x ishould be roughly (cid:16) ( β ∗ − ˆ β ) (cid:62) x
i (cid:17) 2 , and (cid:16) ( β ∗ − ˆ β ) (cid:62) x i [Equation] 2 should be proportional to (cid:107) β ∗ − ˆ β (cid:107) 2given enough samples.
More formally, we will decompose the LHS as
[Equation]
The ﬁrst term contains a (1 − ε ) fraction of uncorrupted random examples sampled from a zero meandistribution with certain tail bound, e.g. sub-exponential for Gaussian and Binomial regression, k -thmoment bound for Poisson regression.
Therefore, we can apply resilience property to bound the ﬁrst term.
Over this work, we heavily utilizethe resilience property of the sample set that is drawn from “nice” distributions.
Take sample meanas an example, resilience [39, 44] dictates that given a large enough sample set S
= { x i } ni =1 , thesample mean of any large enough subset of S will be close to each other.
We deﬁne mean resilienceformally here: Deﬁnition 4.5 (Resilience) . Given a sample set S = { x i } n i =1 , suppose for any T ⊂ S, | T | ≥ (1 − ε ) n ,it holds that (cid:107) 1 | T | (cid:80) i ∈
T x i − 1 | S | (cid:80) i ∈
S x i (cid:107) ≤ τ , then we call the set S satisifes ( ε, τ ) -resilience. Speciﬁcally, under sub-Gaussian distribution, a set S of i.i.d. samples with size n = Ω( d/ε 2 ) satisﬁes( ε, ε (cid:112) log(1 /ε )) resilience with high probability.
Resilience property applies to sub-exponential and k -th moment bounded distribution as well, and this gives us a way to control the behavior of anysubset of good data.
For the second term, we prove that 1 n (cid:80) i ∈ ˆ S ∩ T b ( β (cid:62) x i ) is a strongly convex function again using theresilience property, which implies 1 n (cid:80) [Equation]
[Equation]
Upper bound on the RHS.
To upper bound − 1 n (cid:80) i ∈ ˆ S ∩ E ( y i − b [Equation]
(ˆ β (cid:62) x i ))
( β ∗ − ˆ β ) (cid:62) x i , we will prove an upper bound on (cid:113) 1 n (cid:80) i ∈ ˆ S ∩ E
( y i − b (cid:48)
(ˆ β (cid:62) x i ))
2 and (cid:113) 1 n (cid:80) i ∈ ˆ S ∩ E (( β ∗ − ˆ β ) (cid:62) x i )separately, then apply Cauchy-Schwarz inequality.
The key difﬁculty is bounding (cid:113) 1 n [Equation]
i ∈ ˆ S ∩ E ( y i − b (cid:48) (ˆ β (cid:62) x i )) 2 , as it contains corrupted data points controlled by an adversary, which does not follow any good property possessed by the good stochastic data.
However, since ˆ S contains (1 − ε ) n datapoints with the largest log-likelihood under ˆ β , we can argue that (cid:88) [Equation] −
[Equation]
since otherwise one can replace the examples in ˆ S ∩ E by the ones in T \ ˆ S to form a new setwith better likelihood than ˆ S .
This gives us an upper bound on the negative log-likelihood of y i inˆ S ∩ E .
Therefore we adopt a two step approach to upper bound the (cid:113) [Equation] n [Equation]
First we prove an upper bound on the negative log-likelihood on T \ ˆ S , which becomes a negativelog-likelihood bound on ˆ S ∩ E immediately.
Second we turn the negative log-likelihood bound intoa square error bound.
The two steps vary drastically for different regression models.
For the ﬁrst step of upper bounding thenegative log-likelihood, in Gaussian regression we use the resilience property of the quadratic form ofsub-Gaussian random variable.
In Poisson regression, we leverage the resilience property of y i x i .
In
Binomial regression, since the distribution only has support size m , we directly analyze the resilienceof the negative log-likelihood.
In general GLMs, due to the generality of the likelihood function,we have to again analyze the resilience of the negative log-likelihood directly.
For the second stepof turning the log-likelihood bound to a quadratic bound, we get the bound trivially in Gaussianregression since Gaussian likelihood is indeed quadratic.
For Poisson, Binomial and general GLMssetting, we have to build a proxy function which lower bound the negative log-likelihood function− log f
( y i |(cid:104) ˆ β, x i (cid:105) ) to connect it to the quadratic function.
4.3 Proof for Poisson Regression
As an illustrative example, we show how the above proof sketch can be used for Poisson regression toformally prove Theorem 4.3.
(Proofs for the other GLMs are deferred to Section A in the Appendix)
Lemma 4.6 (Approximate stationary point close to β ∗ for Poisson regression) .
Given a set of datapoints S = { x i , y i } ni =1 generated by a Poisson regression model with ε -fraction of label corruption, and the largest εn labels removed.
Let ˆ β be a max( ε, ε 2 / (cid:107) β ∗ − ˆ β (cid:107) )
-stationary point and (cid:107) ˆ β (cid:107) ≤ R .
Given that n = Ω( dε 2 ) , with probability 0 . 99 , it holds that
[Equation]
Since b (cid:48)(cid:48)
( θ ) =
exp( θ ) is unbounded for Poisson regression, the following lemma (proved in theappendix) shows that alternating minimization algorithm still return an approximate stationary point
Lemma 4.7 (Algorithm 1 ﬁnds an approximate stationary point for Poisson regression) .
Given aset of datapoints S
= { x i , y i } ni =1 generated by a Poisson model with ε c -fraction of corruption.
Assuming that n = Ω( d +log(1 /δ )
ε) , then with probability 1 − δ , the output of Algorithm 1 with inputparameters ε =
ε c , R ≥ (cid:107) β ∗ (cid:107) , η = ε 2 / ( dn ) , is a max( ε, 2 ε 2 (cid:107) β ∗ − ˆ β (cid:107) )
-approximate stationary point.
Proof of Theorem 4.3.
Lemma 4.7 implies the output of Algorithm 1 is a max( ε, ε 2 (cid:107) β ∗ − ˆ β (cid:107) )
approximate stationary point.
Lemma 4.6 then implies that (cid:107) ˆ β − β ∗ (cid:107) =
O ( ε exp( (cid:112) log(1 /ε ))) .
To bound the number of iterations, we need an upper bound on the negative log-likelihood on β = 0 , and a uniformlower bound on the negative log-likelihood.
The initial negative log-likelihood is upper bounded by1 n (cid:80)
i ∈
ˆ S (1) log( y i !)
+ 1 ≤ O ( E [ y 2 i ] + 1) =
O (1) where S (1) contains the smallest (1 − ε ) n labels.
Trivially, there is a 0 lower bound on the negative log-likelihood for Poisson distribution.
Therefore,the algorithm will terminate in dn/ε 2 c iterations.
Result for Sample Corruption Model
The learning problem becomes much harder in the presence of label and covariate corruption, since itbecomes much harder to tell whether a data point is corrupted by simply looking at the likelihood ofy i .
From a technical level, the resilience condition we leveraged on covariate x
i in set E breaks downwhen there is covariate corruption.
Luckily, we are able to restore the resilience property by addinga positive semideﬁnite (PSD) constraint when choosing the set ˆ S ( t ) .
Speciﬁcally, if the covariatedistribution has identity (or known) covariance and sub-Gaussian tail, one can add the PSD constraint1 n (cid:80) w
i x i x (cid:62) i (cid:22) (1 + ε log(1 /ε ))
· I to the ε corrupted data set, and the resulting weighted data set{ w i x i } ni =1 will have the same resilient condition as an uncorrupted data set.
Similar approaches havebeen applied before in robust mean estimation [11] and robust regression [9, 35].
To adapt our alternating minimization algorithm to the sample corruption model, we replace the stepof choosing the best ﬁt subset ˆ S ( t ) by a SDP program, which intuitively tries to ﬁnd the set with thebest likelihood and has bounded covariance at the same time.
This is a covering/packing SDPs whichcan be solved in nearly linear time [34, 2].
5.1 Algorithm
Algorithm
corruption
Input: Set of examples S
= { ( x 1 , y 1 ) , . . .
, ( x n , y n ) } , ε, η, R
Output: ˆ β
3 for ∞ do Let ˆ w ( t ) be the solution of the following SDP
[Equation]
[Equation]
Compute ˆ β ( t +1) =
arg min β, (cid:107) β (cid:107)≤ R (cid:80) ni
=1 − ˆ w ( t ) i log f
( y i |(cid:104) β, x i (cid:105) )
; if (cid:80) ni =1 − ˆ w ( t ) i log f ( y i |(cid:104) ˆ β ( t +1) , x i (cid:105) ) > (cid:80) ni
=1 − ˆ w ( t ) i log f
( y i |(cid:104) ˆ β ( t ) , x i (cid:105) )
− η then Return
( t )
The guarantee of Algorithm 2 is formalized in the following theorems.
Our approach can be extendedto other generalized linear models, as we showed in Appendix B.
Theorem 5.1 (Gaussian regression with sample corruption) .
Given a set of datapoints S = { x i , y i } ni =1 generated by a Gaussian regression model with sub-Gaussian design, ε c -fraction of sample corruption and n = Ω( d +log(1 /δ ) ε) .
With probability 1 − δ , Algorithm 2 with parameters ε = ε c , η = ε 2 c , R = ∞ terminate within O (1 /ε 2 c ) iterations, and output an estimate ˆ β such that
[Equation]
Theorem 5.2 (Binomial regression with sample corruption) .
Given a set of datapoints S = { x i , y i } ni =1 generated by a Binomial regression model with ε c -fraction of sample corruption and n = Ω( d +log(1 /δ ) ε) .
With probability 1 − δ , Algorithm 2 with parameters ε = ε c , η = ε 2 c , R ≥ (cid:107) β ∗ (cid:107) terminate within m/ε 2 c iterations, and output an estimate ˆ β such that
[Equation]
5.2 Proof Sketch
The proof is very similar to the label corruption setting, except that instead of applying the resiliencecondition to a subset of T , we need the same condition on the weighted subset of T .
For examplereplacing
[Equation]
x i .
Luckily, all the resilience conditions apply tothe weighted subset as well.
Compared to the label corruption setting, speciﬁcally there are two places we need to change in theargument.1.
Since x i , i ∈ E is now controlled by the adversary, we can no longer bound(cid:113)
1 n (cid:80) i ∈ ˆ S ∩ E (( β ∗ − ˆ β ) (cid:62) x i ) 2 by the resilience property of sub-Gaussian samples.
Instead,we leverage the fact that corrupted sample with small covariance is also resilient.2.
Previously, T is trivially a feasible solution in the step of choosing ˆ S .
Now we need to argueadditionally that 1 n (cid:80) i ∈
T x i x (cid:62) i (cid:22) (1 + ε log(1 /ε ))
· I
Conclusion
In this paper we provided a general theoretical analysis showing that a simple and practical heuristicnamely the iterative trimmed MLE estimator achieves minimax optimal error rates under labelcorruptions for a wide class of generalized linear models (GLMs).
We also designed algorithmsbased on semi-deﬁnite programming (SDP) to make the estimator robust to covariate corruptionsas well.
While our SDP based algorithm runs in polynomial time, it would be interesting to exploremore practical variants that can handle covariate corruptions.
It would also be interesting to studywhether our techniques can be extended to design robust algorithms for more general exponentialfamilies beyond GLMs.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes] (See Section 6)(c)
Did you discuss any potential negative societal impacts of your work?
[N/A](d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes](b) Did you include complete proofs of all theoretical results?
[Yes]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A](b)
Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[N/A](c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A](d
) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]