Robust Bayesian Regression via Hard Thresholding
Anonymous Author(s)
AffiliationAddress email
Abstract
By combining robust regression and prior information, we develop an effectiverobust regression method that can resist adaptive adversarial attacks.
Due to thewidespread existence of noise and data corruption, it is necessary to recover thetrue regression parameters when a certain proportion of the response variableshave been corrupted.
Methods to overcome this problem often involve robustleast-squares regression.
However, few methods achieve good performance whendealing with severe adaptive adversarial attacks.
Based on the combination ofprior information and robust regression via hard thresholding from [1], this paperproposes an algorithm that improves the breakdown point when facing adaptive adversarial attacks.
Furthermore, to improve the robustness and reduce the estimationerror caused by the inclusion of a prior, the idea of Bayesian reweighting is used toconstruct a more robust algorithm.
We prove the theoretical convergence of proposed algorithms under mild conditions.
Extensive experiments show that, underdifferent dataset attacks, our algorithms achieve state-of-the-art results comparedwith other benchmark algorithms, demonstrating the robustness of the proposedapproach.
Introduction
Least-squares methods are widely used because of their simplicity and ease of operation.
However,due to the inevitable existence of outliers, least-squares methods, such as linear regression, maycause significant bias in practical applications.
Therefore, to meet the challenge of learning reliableregression coefficients in the presence of significant corruption in the response vector, this paperfocuses on robust least-squares regression (RLSR).
RLSR has excellent application value in manyfields, such as signal processing [8][11][25], economics [23], industry [21], biology [12], andcomputer vision [14].
Given a data matrix X =
[ x 1 , ..., x n ] ∈ R d × n , the corresponding response vector y ∈ R n , anda certain number k representing the number of corruptions in the data, the RLSR problem can bedescribed as:
[Equation]
That is, we aim to recover the correct point set S and the regression coefficient w ∗ simultaneously toachieve the minimum regression error.
However, this problem is NP hard, so it is difficult to optimizedirectly [19].
To solve the above problem, a commonly used data generation model is y =
X T w ∗ + b ∗ + ϵ ,where w ∗ is the true regression coefficient we wish to recover and ϵ is a dense white noise vectorsubject to a specific distribution, that is, ∥ ϵ ∥ 0 ∼ n .
The vector b ∗ is k -sparse, which means thatthere are only k non-zero values, representing k unbounded noise terms in the response vector.
Afteryears of development, there are many ways to find a reasonable solution to the problem in Eq.
However, these methods typically only achieve good performance under specific conditions.
Themain challenge is the low breakdown point of conventional methods.
The breakdown point k is ameasure of robustness, which means the number of corruptions that the RLSR algorithm can tolerate.
We can express k as the proportion of all data points: k
= α · n .
Many RLSR algorithms cannotguarantee theoretical convergence as the value of k increases.
For example, McWilliams et al.
[13]used weighted subsampling for linear regression, but only had a breakdown point of α
= O (1 / √ d ) .
Prasad et al.
[18] proposed a robust gradient estimator that can be applied to linear regression, but theirmethod only tolerates corruption up to α
= O (1 / log d ) .
Other methods may have a higher breakdownpoint, but tend to assume a specific pattern of data corruption.
One representative adversary model forintroducing data corruption is the oblivious adversarial attack (OAA), in which the opponent generatesk sparse vectors while completely ignoring X , w ∗ , and ϵ .
The work of Bhatia et al.
[1] and Suggalaet al.
[20] reported excellent results against OAAs by using a novel hard thresholding method; indeed,[20] suggested that α may even get close to 1 as n → ∞ .
The recent online fast robust regressionalgorithm [16] also has consistent convergence with a mild condition by using Stochastic Gradient Decent (SGD) algorithm.
However, these methods cannot resist adaptive adversarial attack (AAA),in which opponents can view X , w ∗ , and ϵ before determining b ∗ .
Handling AAA is a challengingtask, and so many methods can only guarantee a very low breakdown point, especially when the datadistribution is not normal [4][9][15].
Bhatia et al.
[2] proposed the thresholding operator-based robustregression method and their breakdown point reach 1 / 65 for noiseless model i.e., ϵ ≡ 0 .
However,their method can give a consistent estimation only in noiseless case.
Karmalka et al.
[10] had agood result in sparse robust linear regression by applying the L 1 regression and the breakdown pointof their method reaches 0.239, but their estimation is consistent only when white noise ϵ is sparse.
Diakonikolas et al.
[6] considered the situation that X and y may have outliers simultaneously, andproposed a filter algorithm in which the error bound is O
( α log(1 /α ) σ ) .
However, their methodrequires accurate data covariance of the true data distribution or numerous unlabeled correct data toestimate the data covariance, which are often unavailable in practice.
The limitations of the above-mentioned methods can be attributed to a lack of prior knowledge fromthe real data, making it difficult to distinguish the set of correct points in the case of AAAs.
Gülçehreet al.
[7] showed that prior information effectively improves the accuracy of machine learning.
Inmany application scenarios in industry, economics, and biology, prior knowledge such as previousexperimental data or engineering data are available.
The goal of this paper is to propose a new robustregression that can integrate available prior information, even if the prior information is not veryaccurate.
The typical approach for integrating prior information is the Bayesian method.
This provides a wayof formalizing the process of learning from data to update beliefs in accordance with recent notionsof knowledge synthesis [5].
However, generic Bayesian method is also sensitive to outliers.
Thusrobust Bayesian method should be considered to produce more reliable estimates in the presence ofdata corruption.
Polson et al.
[17] used the local variance to assign each point a local parameter thatmakes the estimation result robust.
Furthermore, Wang et al.
[22] proposed a local parameterizationmethod and used empirical Bayesian estimation to determine the global parameters.
Bhatia etal.
[3] proposed a Bayesian descent method using an unadjusted Langevin algorithm (ULA), whichguarantees convergence in a finite number of steps.
Wang et al.
[24] employed Bayesian reweightingto assign different weights to samples, thus reducing the impact of outliers.
In this paper, we combine the Bayesian method with a hard thresholding method [1] and propose twoalgorithms, which we call TRIP and BRHT.
Through assigning a simple normal prior on the coefficients, TRIP can significantly increase the breakdown point when resisting AAAs.
To further improvethe accuracy of estimation, we propose BRHT algorithm through applying Bayesian reweightingmethod [24] to coefficient estimation.
Experiments show that BRHT is even more resistant to AAAsand gives lower estimation errors, demonstrating that our method achieves significantly improvedrobustness.
Our Contributions : The main contribution of this paper is proposing new methods that combiningthe prior and robust regression to increase the breakdown point when encountering AAAs.
We derivethe theoretical guarantees given by our proposed algorithms.
Compared with the consistent robustregression (CRR) algorithm proposed in [1], we prove that our algorithms guarantee convergenceunder a weaker condition, which also shows that our methods improves the breakdown point.
We alsoestablish an extended experiment to test the effectiveness of the algorithms.
Compared with other basic algorithms, the experimental results show that our methods significantly outperform alternativemethods under AAAs.
Moreover, BRHT algorithm is also competitive against OAAs.
Paper Organization : We state the problem formulation and present some notation and tools in
Section 2.
In Section 3, we describe the details of our proposed TRIP and BRHT algorithms.
Thetheoretical properties of these two algorithms are discussed in Section 4.
Section 5 presents extensiveexperimental results that demonstrate the excellent performance of our proposed algorithms.
Section6 concludes this paper.
Problem Formulation
In this study, we mainly focus on the problem of RLSR under AAAs.
We are given a covariant matrix
X =
[ x 1 , ..., x n ] ∈ R d × n , where x i ∈ R d .
The true coefficient of the regression model is denotedby w ∗ .
The response vector y ∈ R n is generated by:
[Equation]
The perturbations to the response vector consist of two parts: the adversarial corruption vectorintroduced by b ∗ , which is a k -sparse vector, and the dense white noise ϵ i ∼ N (0 , σ 2 ) .
Our goal isto recover the true regression coefficient w ∗ while simultaneously determining the corruption set S .
To illustrate our problem, we first pay attention to the standard robust regression problem in Eq.
From the viewpoint of probability, the problem can be transformed into a log-likelihood version:
[Equation]
We will try to convert the problem in Eq.
(3) into a Bayesian version.
From the Bayesian viewpoint,we consider p w ( w ) as the prior of the coefficients in the model.
In addition, we add the localizationparameter r and its prior p r ( r ) to reflect the change introduced by each additional sample.
For anysubset S ⊆ [ n ] , the distribution of all parameters and data X S , y S is:
[Equation]
The posterior distributions p ( w , r S | X S , y S ) and p ( y S , w , r S | X S ) differ by a regularization constant.
We ignore this regularization constant in the posterior distribution and only consider the main termsof the parameters.
We then formulate the Bayesian RLSR problem of searching for the subset andcoefficients by maximizing the log-posterior:
[Equation]
Note that we do not add any prior on σ 2 and only treat this as an adjustable parameter.
This isbecause, in the initial stage of the algorithm described in Section 3, the estimated σ 2 will be largedue to the existence of outliers, and this will make the estimation of w excessively biased to the priordistribution.
This bias will be harmful, especially when the prior is not sufficiently accurate.
Thisphenomenon can be observed in Section 3.1.
To prove the positive effect of the prior on the RLSRproblem, we require the properties of Subset Strong Convexity (SSC) and Subset Strong Smoothness(SSS) .
Given a set S ⊂ [ n ] , X S :
=
[ x i ∈ S ] ∈ R d ×| S | signifies the matrix with columns in the set S .
The smallest and largest eigenvalues of a square symmetric matrix X are denoted by λ min ( X ) andλ max ( X ) .
Definition 1 (SSC Property) .
A matrix X ∈ R
d × n is said to satisfy the SSC property at level m withconstant λ m if the following holds:
[Equation]
Definition 2 (SSS Property) .
A matrix X ∈ R
d × n is said to satisfy the SSS property at level m withconstant Λ m if the following holds:
[Equation]
These two properties are proposed in [2], and are intended to standardize the generation of the datamatrix so that it will not be too abnormal.
They are used to prove the theorems in Section 4.
Input: Covariates X =
[ x 1 , ..., x n ] , responses y =
[ y 1 , ..., y n ] T , prior knowledge w 0 , penalty matrix M , corruption index k , tolerance ϵ
Output: solution ˆ w 1: b 0 ← 0 , t ← 0 ,
P MX ← X T ( XX T + M ) − 1 X , P MM ← X T ( XX T + M ) − 1 M 2: while ∥ b t − b t − 1 ∥ 2 > ϵ do 3: b t +1 ← HT k
( P MX b t
+ ( I − P MX )
y − P MM w 0 ) 4: t ← t + 1 ; 5: end while 6: return ˆ w ← ( XX T ) − 1 X
( y − b t )
Methodology
We first ignore the localization parameter r in Eq.
(5) and propose a simple method called TRIP in
Section 3.1.
TRIP demonstrates the effect of a prior on the hard thresholding method.
To improve therobustness and the accuracy of the estimation, the BRHT algorithm is proposed in Section 3.2.
3.1 TRIP: Hard Thresholding Approach to Robust Regression with Simple Prior
We propose a robust regression algorithm called TRIP (Algorithm 1), a hard Thresholding approachto Robust regression with sImple Prior.
In this subsection, only the prior p w ( w ) is considered andthe localization parameter r is not added to the model.
We assume the variance σ 2 of ϵ i can be setby ourselves and that the prior p w ( w ) obeys a normal distribution N ( w 0 , Σ 0 ) , where w 0 and Σ 0are determined in advance.
Through the above simple parameter settings, the problem in Eq.
(5) istransformed into the problem in Eq.
(1) with an additional regularization term:
[Equation]
where M = (Σ 0 /σ 2 ) − 1 .
To solve this problem, we are motivated by the hard thresholding methodproposed by Bhatia [1], which concentrated on recovering the errors instead of selecting the ‘cleanest’set.
The problem in Eq.
(8) can be formulated as min w ∈ R p , ∥
b ∥ 0 ≤ k ∗ 12 ∥ X T w − ( y − b ) ∥ 22 + 12 ( w −w 0 ) T M ( w − w 0 ) .
Thus, if we have an estimation ˆ b of the corruption vector b ∗ , the estimationof w ∗ can be easily obtained by ˆ w = ( XX T + M ) − 1 [ X ( y − ˆ b ) + M w 0 ] .
By substituting thisestimation into the optimization problem, we obtain a new formulation of the problem:
[Equation]
where P MX = X T ( XX T + M ) − 1 X , P MM = X T ( XX T + M )
− 1 M .
The hard thresholdingstep in the TRIP algorithm can be viewed as b t +
=
HT k ( b t − ∇ f ( b t )) , where k is the selectedcorruption coefficient.
The hard thresholding operator HT k is defined as follows.
Definition 3 (Hard Thresholding) .
For any vector r ∈ R n , let δ − 1 r ( i ) represent the position of thei th element in r , which are arranged in descending order of magnitude.
Then, for any k < n , thehard thresholding operator is defined as ˆ r =
HT k
( r ) , where ˆ r i =
r i if δ − 1 r ( i ) ≤ k and 0 otherwise.
The difference between the proposed TRIP algorithm and the original CRR [1] is the form of iterationstep. The iteration step in both TRIP and CRR can be expressed uniformly as HT k ( y − X T w t ) , butw t = ( XX T + M ) − 1 [ X ( y − b t ) + M w 0 ] in TRIP and w t =
( XX T ) − 1 X
( y − b t ) in CRR.
Thew t in CRR is just a simple least square estimation, while the prior added in TRIP can be regardedto adding a quadratic regularization in each iteration. This quadratic regularization can avoid thecandidate of iteration that is too far from the prior mean, which is also helpful to ensure the numericalstability of solution. Thus, as long as the prior is not mis-specified too much, TRIP will be morelikely to identify the uncorrupted points, and the final result of TRIP will be more robust than CRR. Therefore, the prior plays an important role in TRIP.
However, the weight of a prior in the solutiondepends entirely on the matrix M =
(Σ 0 /σ 2 ) − 1 .
Therefore, if we use an estimation of ˆ σ 2 to replace
Input: Covariates X =
[ x 1 , ..., x n ] , responses y =
[ y 1 , ..., y n ] T , prior distribution p r ( r ) , p w ( w ) , corruption index k , tolerance ϵ
Output: solution ˆ w 1: b 0 ← 0 , t ← 0 , 2: while ∥ b t − b t − 1 ∥ 2 > ϵ do 3: w t ← V BEM ( X, y − b t , p r ( r ) , p w ( w )) 4: b t +1 ← HT k
( y − X T w t ) 5: t ← t + 1 ; 6: end while 7: return ˆ w ← ( XX T ) − 1 X ( y − b t ) σ 2 , or give σ 2 a prior to calculate its posterior distribution, the overestimation of σ 2 will cause asevere increase in M in the initial iteration steps. This will mislead the iteration and cause somedeviation in the final results. To overcome this difficulty we can directly treat M as an adjustableparameter to control by specifying the form such as M = sI , where s is a positive number, and thesuitable parameter can be chosen through 5-fold or 10-fold cross validation.
3.2 BRHT: Robust Bayesian Reweighting Regression via Hard Thresholding
In this subsection, we describe how the Bayesian reweighting method is combined with hard thresholding to give a more robust algorithm, BRHT (Algorithm 2), a robust Bayesian Reweighting regressionvia Hard Thresholding.
We first introduce the reweighted probabilistic model (RPM) proposed in[24] for traditional linear regression.
For the covariates X and the response y , the RPM model canbe formulated as follows:
[Equation]
where r is the local weight assigned to each sample, Z is the normalizing constant, ℓ ( y i | w , x i , σ 2 )represents the likelihood of the normal distribution N ( x Ti w , σ 2 ) , and p w ( w ) , p r ( r ) are the priorsof w and r , respectively.
By ignoring the normalizing constant, the problem in Eq.
(5) can betransformed into the following form under this RPM setting:
[Equation]
The specific form of the prior p r ( r i ) can be set to any nonnegative random variable distribution,including (but not limited to) the Gamma distribution, Beta distribution, or log-normal distribution.
Here, we still use the normal distribution N ( w 0 , Σ 0 ) as the form of p w ( w ) .
To solve the optimization problem in Eq.
(11), we use the two-step BRHT algorithm.
The keyiteration step in BRHT is b t +1 ← HT k ( y − X T w t ) , where w t is calculated by maximizing thelog-posterior of the RPM model:
[Equation]
However, the direct inference of Eq.
(12) is hard because of the nonconvexity of this problem.
Ingeneral, the parameters in this RPM model can be divided into two parts: the global variable w and the local latent variable r .
To solve the inference problem of RPM, a feasible method is to usevariational Bayesian expectation maximization (VBEM).
We set q ( w , r ) =
q ( w ) q ( r ) to approximatethe true posterior after several iterations of VBEM (Algorithm 3), and replace the estimation of wby the maximum a posteriori (MAP) estimation from the final approximate posterior.
Full detailsof the VBEM method are given in Appendix A.
It is reasonable to ask why we are using Bayesianreweighting.
Note that the iteration step in the TRIP algorithm is b t
+1 ← HT k
( P MX b t
+
( I −
P MX )
y − P MM w 0 )
=
HT k ( y − X T w t ) , where w t =
( XX T + M ) − 1 [ X ( y − b t )
+ M w 0 ] .
Although we can show that TRIP already guarantees theoretical convergence, the estimation ofw ∗ in every iteration still uses least-squares with a penalty term, which is easily affected by thecorrupted points.
This disadvantage forces us to assign a higher weight to the prior to resist severedata corruption in the case of AAAs.
However, a higher weight on the prior means a larger estimationbias.
When applying the Bayesian reweighting method, the estimation in each step is more robustthan the least-squares result, and thus the weight on the prior can be reduced to guide the iteration.
Therefore, the estimation bias is relatively small and the results are more robust.
This is reflected inthe experimental results presented in Section 5.
We should also explain why we only use a prior for a few parameters.
It is important to ensurethat the prior weights are neither too high nor too low.
As mentioned earlier, if we treat σ 2 as theparameter to be estimated, this places too much weight on the prior.
We also ensure that p w ( w ) doesnot create more uncertainty, such as setting p w ( w ) to p ( w , α ) =
N ( w 0 , α − 1 Σ 0 ) Gam ( α |
a α , b α ) .
Data corruption means that the subset of the training data may vary greatly from the prior information.
Thus, when calculating the posterior, the variance of w controlled by α will be very large to fit the data,and so the prior information w 0 will have lower weights in the inference step.
The above problemsalso mislead the estimation and the selection of subsets, so we do not consider the uncertainty ofthese quantities and simply treat them as model parameters to be set in advance.
An adjustmentmethod for all the parameters in BRHT is described in Appendix D.
Theoretical Convergence Analysis
In this section, we establish the convergence theory for the TRIP algorithm, and clearly explain howthe prior effectively enhances the convergence of the RLSR model.
Theorems 1 and 2 summarizethe results.
We also show the theoretical guarantee of the BRHT algorithm in Theorems 3–5, whichfurther demonstrate the special properties achieved by using Bayesian reweighting.
Before presentingthe convergence result, we first introduce some notation.
Let λ t :=
( XX T + M ) − 1 X ( b t − b ∗ ) ,g :=
( I − P MX ) ϵ , and f :=
P MM ( w ∗ − w 0 ) .
Let S t :=
[ n ] \ supp ( b t ) be the chosen subset that isconsidered to be uncorrupted, and I t :=
supp ( b t ) ∪ supp ( b ∗ ) .
Theorem 1.
Let X =
[ x 1 , . . .
, x n ] ∈ R d × n be the given data matrix and y =
X T w ∗ + b ∗ + ϵbe the corrupted output with sparse corruption of ∥ b ∗ ∥ 0 ≤ k · n .
For a specific positive semidefinite matrix M , X satisfies the SSC and SSS properties such that
[Equation]
) < 1 .
Then, ifk > k ∗ , it is guaranteed with a probability of at least 1 − δ that, for any ε, δ > 0 , ∥ b T 0 − b ∗ ∥ 2 ≤ε + O ( e 0 ) + O (
[Equation]
{+T 
 =+} O (log(
[Equation]
[-∥ w ∗ − w 0 ∥ 2 after T 
-]
)) iterations of TRIP, wheree [-
-]0 =
O ( σ (cid:113) ( k + k ∗ ) log n δ ( k + k ∗ ) ) under the normal design.
Theorem 2.
Under the conditions of Theorem 1, and assuming that x i ∈ R d are generated from thestandard normal distribution, if k > k ∗ , it is guaranteed with a probability of at least 1 − δ that,for any ε, δ > 0 , the current estimation coefficient w T 0 satisfies ∥ w T 0 − w ∗ ∥ 2 ≤ O ( 1 √ n )( ε + e 0 ) +
O ( √ k + k ∗ λ max ( M ) n 3 /)
∥ w ∗ − w 0 ∥ 2 after T
= O (log( ∥ b ∗ ∥ 2 ε )) steps.
For positive semi-definite matrices XX T and M , λ min ( XX T + M ) ≥ λ min ( XX T ) + λ min ( M ) .
Thus, the condition [Equation] in Theorem 1 is weaker than the condition
[Equation]
of Lemma 5 of Bhatia [1], which shows that a prior can effectively improve the convergence of thealgorithm.
Assigning a higher weight to a prior means that M has larger eigenvalues, so that theconvergence condition will be more easily satisfied.
As a result, the TRIP algorithm can tolerate ahigher proportion of outliers than the CRR method of Bhatia [1] and achieves a higher breakdownpoint.
In fact, under the condition lim n →∞ λ min ( M ) n = ξ , we can give an approximate expression ofthe breakdown point for TRIP when ξ is not too large: k ∗ ≤ k ≤ (0 .
3023 − √ 0 .
0887 − 0 .
0040 ξ ) n .
Details can be found in Appendix ?? .
However, the improved convergence comes at the cost ofan unavoidable reduction in precision.
This can be seen from Theorem 2.
If the data corruption issuch that k ∗ is O ( n ) and the maximum eigenvalue of M is also O ( n ) , then the bias of ˆ w cannot bedecreased by adding more samples, which shows that there is a trade-off between convergence andaccuracy.
A reliable prior improves both accuracy and convergence because it has a higher weight.
However, an inaccurate prior can also be helpful as long as it is quite different from the distributionof outliers, and the convergence can be improved through a prior with a low weight.
To prove the properties of our BRHT algorithm, we define the following two intermediate variablesto simplify the description:
[Equation]
Theorem 5.
Suppose that the prior of r i is independently and identically distributed (iid).
We considerthe t th iteration step of the BRHT algorithm, where w t , r t = arg max w ∈ R d , r ∈ R n + M ( w , r , b t )and b t =
HT k
( y − X T w t − 1 ) is obtained from the hard thresholding step.
Then, we have that
U ( w t , r t , S t +1 ) ≥ U ( w t − 1 , r t − 1 , S t ) .
Theorem 6.
Consider a data matrix X and a specific positive semi-definite matrix M satisfyingthe SSC and SSS properties such that
[Equation]
) < 1 .
Then, there exist α > 0 and 0 <γ ≤ 1 + ϵ , where ϵ is a small number, such that if k > k ∗ and Σ in the prior p w ( w ) is ασ 2 M − 1 ,it is guaranteed with a probability of at least 1 − δ that, for any ε, δ > 0 , ∥ b T 0 − b ∗ ∥ 2 ≤ ε + O ( e 0 ) + O (
[Equation]
γ ∥ w
[Equation]
− w
[Equation]
∥ 2 after T 0 =
O (log(
[Equation]
)) iterations of BRHT, wheree 0 =
O ( σ (cid:113) ( k + k ∗ ) log n δ ( k + k ∗ ) ) under the normal design.
Theorem 7.
Under the conditions of Theorem 4 and with x i ∈ R d generated from the standardnormal distribution, there exist α > 0 and 0 < γ ≤ 1 + ϵ , where ϵ is a small number, such thatif k > k ∗ and Σ in the prior p w ( w ) is ασ 2 M − 1 , it can be guaranteed with a probability of atleast 1 − δ that, for any ε, δ > 0 , the current estimation coefficient w T 0 satisfies ∥ w T 0 − w ∗ ∥ 2 ≤
[Equation]
steps
[Equation]
Theorem 5 shows that our BRHT algorithm is reasonable because it optimizes the problem in Eq.(11) in each step.
Theorems 6 and 7 guarantee the convergence of the parameter, which means thatif we introduce a prior p w ( w ) =
N ( w 0 , Σ 0 ) to the TRIP algorithm, it is convergent.
There thenexists some α > 0 such that the prior p w ( w ) =
N ( w 0 , α Σ 0 ) in the BRHT algorithm guaranteesconvergent parameters and the bias of the estimation of w ∗ will be γ times that of TRIP.
Note thatα is usually relatively large in practice, which causes a lower prior weight in BRHT.
Thus, whenthe convergence can be guaranteed, the bias of the estimator ˆ w can be significantly reduced because
BRHT assigns a lower weight to the prior.
Even if the data are seriously corrupted, BRHT can ensuregood results without significant error.
All proofs of these theorems are given in Appendix C.
Experiments
In this section, we first consider how to effectively ‘corrupt’ the dataset using two different attacks:
OAA and AAA.
We then report an extensive experimental evaluation to verify the robustness of theproposed methods.
5.1 Data and Metrics
In our experiments, the data generation can be divided into two steps.
First, we generate the basicmodel.
The true coefficient w ∗ is chosen to be a random unit norm vector.
The covariant x i are iid in N (0 , I d ) .
The data are generated by y i =
x Ti w ∗
+ ϵ i , where ϵ i are iid in N (0 , σ 2 ) .
We set σ = 1 inthe experiments.
The second step is to generate the corrupted data using two kinds of attacks: OAAand AAA, as described in Section 5.2.
The aim is to produce k ∗ corrupted responses in the wholedataset.
The prior coefficient w 0 is generated by w ∗ + ν u , where u is a random unit norm vector Input: Covariates X = [ x 1 , ..., x n ] , responses y =
[ y 1 , ..., y n ] T , true parameter w ∗ penalty coefficient δ , corruption index k , tolerance ϵ
Output: solution ˆ w 1: b 0 ← 0 , t ← 0 , P δX ← X T ( XX T − δI ) − 1 X , P δ ← X T ( XX T − δI ) − 1 δI 2: while ∥ b t − b t − 1 ∥ 2 > ϵ do 3: b t +1 ← HT k ( P δX b t + ( I − P δX )
y + P δ w ∗ ) 4: t ← t + 1 ; 5: end while 6: ˆ w ← ( XX T ) − 1 X ( y − b t ) 7: C ← supp ( b t ) 8: return y C
= X TC ˆ w and ν is a non-negative number ( ν is set to 0.5 unless otherwise stated). Σ 0 takes the form sI , wheres takes a different value for each method.
All parameters are fixed in each experiment.
Following the setting in [1], we measured the performance of the regression coefficients by thestandard L 2 error: r ˆ w = ∥
ˆ w − w ∗ ∥ 2 .
To judge whether the algorithm had converged, we used thetermination criterion ∥ w t +1 − w t ∥ 2 ≤ 10 − 4 .
All results were averaged over 10 runs.
5.2 Corruption Methods
To demonstrate the efficiency of our proposed methods, we apply two different attacks to the dataset:
OAA and AAA.
The details of these two attacks are shown as follows
OAA : The set of corrupted points S is selected as a uniformly random k -sized subset of [ n ] , and thecorresponding response variables are set as y i
=
x Ti w ∗ + b i
+ ϵ i , where b i are sampled from theuniform distribution U [0 , 10] and the white noise ϵ i ∼ N (0 , σ 2 ) .AAA
: We use all information from the true data distribution to corrupt the data, and propose anadaptive data corruption algorithm (ADCA).
This algorithm is quite similar to TRIP; full details of
ADCA (Algorithm 4) are given in Appendix B. δ is set to 0 .
1 n for n = 1000 , p = 200 , and to 0 .
2 nfor n = 2000 , p = 100 .
Both TRIP and BRHT employ the prior p w ( w ) while the variance Σ 0 of p w ( w ) in OAA is set tobe four times higher than that in AAA, which means that the prior has a higher weight in AAA.
Other parameters for these two algorithms are fixed.
The prior distribution p r ( r ) is set to the Gammadistribution unless otherwise stated.
We also design another leverage point attack (LPA) to test the robustness of our methods.
Moreresults can be seen in Appendix ?? .
5.3 Methods Comparison
Our methods are compared with three baselines: 1) CRR [1] is an effective robust regression methodin cases where there are large numbers of random outliers; 2) Reweighted robust Bayesian regression(RRBR) [24] allows us to judge whether our proposed methods are better than the original method;3)
Rob-ULA [3] is an effective robust Bayesian inference method that approximately converges tothe real posterior distribution in a finite number of steps in the presence of outliers.
The parametersof RRBR are the same as in the BRHT algorithm, except for the hard thresholding part.
For morecomparisons of other methods, please see Appendix ?? .
5.4 Recovery Properties of Coefficients and Uncorrupted Sets
CRR, RRBR, and Rob-ULA are excellent robust regression methods or Bayesian inference methods,but they all show their limitations in the face of different kinds of attacks.
CRR achieves the bestperformance in the face of OAAs because it is theoretically unbiased, but it collapses rapidly whenfacing AAAs, as shown in Figures 1(c) and 1(d).
RRBR and Rob-ULA take priors into consideration,but RRBR cannot resist AAAs and Rob-ULA produces poor results under OAAs, as shown in Figures [Figure] [Figure] 1(a)–1(d).
TRIP produces a good effect against AAAs, while BRHT is not only optimal against
AAAs, but also displays a similar effect to CRR in the case of OAAs.
This shows that BRHT is themost robust algorithm among those compared in this experiment.
The TRIP and BRHT algorithms are compared in Figures 2(a) and 2(b).
The TRIP method incorporates too much prior information in the case of OAAs, resulting in a greater estimation error thanthose of BRHT and CRR as shown in Figure 2(a).
However, under AAAs, the prior information of
TRIP and BRHT is enhanced by a factor of four, as described in Section 5.2.
BRHT converges undera weaker prior, while TRIP becomes trapped around a local optimum.
This shows that BRHT onlyneeds to integrate weak prior information to ensure convergence.
Figures 2(c) and 2(d) illustrate theconvergence properties under different weights of the prior p r ( r ) and coefficient prior p w ( w ) .
Figure2(c) shows that BRHT is not especially sensitive to the weight prior p r ( r ) when this prior is relativelyreliable.
A log-normal distribution is the best choice when the prior p w ( w ) is relatively close to thereal parameters and a Gamma distribution is more robust when the prior is imprecise, as shown in
Figure 2(d).
Conclusion
This paper has described a novel robust regression algorithm named TRIP that achieves strong resultsin terms of resisting AAAs.
By adding a prior to the robust regression via hard thresholding, therecovery of coefficients is significantly improved.
Another algorithm, named BRHT, was designedto improve the robustness of TRIP and reduce the estimation error through the use of Bayesianreweighting regression.
We prove that both algorithms have strong theoretical guarantees and that thealgorithms converge linearly under a mild condition.
Extensive experiments have illustrated that ouralgorithms outperform benchmark methods in terms of both robustness and efficiency.
There are several interesting future directions to extend current work.
Firstly, in this article, we onlyconsider the case when y is corrupted.
One would consider using the prior information to better dealwith the problem where both y and X are corrupted.
Secondly, it would be also interesting to furtherreduce the effect of a prior on the estimation to make it consistent.