Robust Bayesian Regression via Hard Thresholding
Abstract
By combining robust regression and prior information, we develop an effective robust regression method that can resist adaptive adversarial attacks.
Due to the widespread existence of noise and data corruption, it is necessary to recover the true regression parameters when a certain proportion of the response variables have been corrupted.
Methods to overcome this problem often involve robust least-squares regression.
However, few methods achieve good performance when dealing with severe adaptive adversarial attacks.
Based on the combination of prior information and robust regression via hard thresholding from [1], this paper proposes an algorithm that improves the breakdown point when facing adaptive adversarial attacks.
Furthermore, to improve the robustness and reduce the estimation error caused by the inclusion of a prior, the idea of Bayesian reweighting is used to construct a more robust algorithm.
We prove the theoretical convergence of proposed algorithms under mild conditions.
Extensive experiments show that, under different dataset attacks, our algorithms achieve state-of-the-art results compared with other benchmark algorithms, demonstrating the robustness of the proposed approach.
Introduction
Least-squares methods are widely used because of their simplicity and ease of operation.
However, due to the inevitable existence of outliers, least-squares methods, such as linear regression, may cause signiﬁcant bias in practical applications.
Therefore, to meet the challenge of learning reliable regression coefﬁcients in the presence of signiﬁcant corruption in the response vector, this paper focuses on robust least-squares regression (RLSR).
RLSR has excellent application value in many ﬁelds, such as signal processing [8][12][26], economics [23], industry [21], biology [13], remote sensing [11] and intelligent transportation [25].
Given a data matrix X =
[ x 1 , ..., x n ] ∈ R d × n , the corresponding response vector y ∈ R n , and a certain number k representing the number of corruptions in the data, the RLSR problem can be described as:
[Equation]
That is, we aim to recover the correct point set S and the regression coefﬁcient w ∗ simultaneously to achieve the minimum regression error.
However, this problem is NP hard, so it is difﬁcult to optimize directly [19].
To solve the above problem, a commonly used data generation model is y =
X T w ∗ + b ∗ + ϵ , where w ∗ is the true regression coefﬁcient we wish to recover and ϵ is a dense white noise vector subject to a speciﬁc distribution, that is, ∥ ϵ ∥ 0 ∼ n .
The vector b ∗ is k -sparse, which means that there are only k non-zero values, representing k unbounded noise terms in the response vector.
After years of development, there are many ways to ﬁnd a reasonable solution to the problem in Eq.
However, these methods typically only achieve good performance under speciﬁc conditions.
The main challenge is the low breakdown point of conventional methods.
The breakdown point k is a measure of robustness, which means the number of corruptions that the RLSR algorithm can tolerate.
We can express k as the proportion of all data points: k
= α · n .
Many RLSR algorithms cannot guarantee theoretical convergence as the value of k increases.
For example, McWilliams et al.
[14] used weighted subsampling for linear regression, but only had a breakdown point of α
= O (1 / √ d ) .
Prasad et al.
[18] proposed a robust gradient estimator that can be applied to linear regression, but their method only tolerates corruption up to α
= O (1 / log d ) .
Other methods may have a higher breakdown point, but tend to assume a speciﬁc pattern of data corruption.
One representative adversary model for introducing data corruption is the oblivious adversarial attack (OAA), in which the opponent generates k sparse vectors while completely ignoring X , w ∗ , and ϵ .
The work of Bhatia et al.
[1] and Suggala et al.
[20] reported excellent results against OAAs by using a novel hard thresholding method; indeed, [20] suggested that α may even get close to 1 as n → ∞ .
The recent online fast robust regression algorithm [16] also has consistent convergence with a mild condition by using Stochastic Gradient Decent (SGD) algorithm.
However, these methods cannot resist adaptive adversarial attack (AAA), in which opponents can view X , w ∗ , and ϵ before determining b ∗ .
Handling AAA is a challenging task, and so many methods can only guarantee a very low breakdown point, especially when the data distribution is not normal [4][9][15].
Bhatia et al.
[2] proposed the thresholding operator-based robust regression method and their breakdown point reach 1 / 65 for noiseless model i.e., ϵ ≡ 0 .
However, their method can give a consistent estimation only in noiseless case.
Karmalka et al.
[10] had a good result in sparse robust linear regression by applying the L 1 regression and the breakdown point of their method reaches 0.239, but their estimation is consistent only when white noise ϵ is sparse.
Diakonikolas et al.
[6] considered the situation
that
X
and y may have outliers simultaneously, and proposed a ﬁlter algorithm in which the error bound is O
( α log(1 /α ) σ ) .
However, their method requires accurate data covariance of the true data distribution or numerous unlabeled correct data to estimate the data covariance, which are often unavailable in practice.
The limitations of the above-mentioned methods can be attributed to a lack of prior knowledge from the real data, making it difﬁcult to distinguish the set of correct points in the case of AAAs.
Gülçehre et al.
[7] showed that prior information effectively improves the accuracy of machine learning.
In many application scenarios in industry, economics, and biology, prior knowledge such as previous experimental data or engineering data are available.
The goal of this paper is to propose a new robust regression that can integrate available prior information, even if the prior information is not very accurate.
The typical approach for integrating prior information is the Bayesian method.
This provides a way of formalizing the process of learning from data to update beliefs in accordance with recent notions of knowledge synthesis [5].
However, generic Bayesian method is also sensitive to outliers.
Thus robust Bayesian method should be considered to produce more reliable estimates in the presence of data corruption.
Polson et al.
[17] used the local variance to assign each point a local parameter that makes the estimation result robust.
Furthermore, Wang et al.
[22] proposed a local parameterization method and used empirical Bayesian estimation to determine the global parameters.
Bhatia et al.
[3] proposed a Bayesian descent method using an unadjusted Langevin algorithm (ULA), which guarantees convergence in a ﬁnite number of steps.
Wang et al.
[24] employed Bayesian reweighting to assign different weights to samples, thus reducing the impact of outliers.
In this paper, we combine the Bayesian method with a hard thresholding method [1] and propose two algorithms, which we call TRIP and BRHT.
Through assigning a simple normal prior on the coefﬁcients, TRIP can signiﬁcantly increase the breakdown point when resisting AAAs.
To further improve the accuracy of estimation, we propose BRHT algorithm through applying Bayesian reweighting method [24] to coefﬁcient estimation.
Experiments show that BRHT is even more resistant to AAAs and gives lower estimation errors, demonstrating that our method achieves signiﬁcantly improved robustness.
Our Contributions : The main contribution of this paper is proposing new methods that combining the prior and robust regression to increase the breakdown point when encountering AAAs.
We derive the theoretical guarantees given by our proposed algorithms.
Compared with the consistent robust regression (CRR) algorithm proposed in [1], we prove that our algorithms guarantee convergence under a weaker condition, which also shows that our methods improves the breakdown point.
We also establish an extended experiment to test the effectiveness of the algorithms.
Compared with other basic algorithms, the experimental results show that our methods signiﬁcantly outperform alternative methods under AAAs.
Moreover, BRHT algorithm is also competitive against OAAs.
Paper Organization : We state the problem formulation and present some notation and tools in
Section 2.
In Section 3, we describe the details of our proposed TRIP and BRHT algorithms.
The theoretical properties of these two algorithms are discussed in Section 4.
Section 5 presents extensive experimental results that demonstrate the excellent performance of our proposed algorithms.
Section 6 concludes this paper.
Problem Formulation
In this study, we mainly focus on the problem of RLSR under AAAs.
We are given a covariant matrix X =
[ x 1 , ..., x n ] ∈ R d × n , where x i ∈ R d .
The true coefﬁcient of the regression model is denoted by w ∗ .
The response vector y ∈ R n is generated by:
[Equation]
The perturbations to the response vector consist of two parts: the adversarial corruption vector introduced by b ∗ , which is a k -sparse vector, and the dense white noise ϵ i ∼ N (0 , σ 2 ) .
Our goal is to recover the true regression coefﬁcient w ∗ while simultaneously determining the corruption set S .
To illustrate our problem, we ﬁrst pay attention to the standard robust regression problem in Eq.
From the viewpoint of probability, the problem can be transformed into a log-likelihood version:
[Equation]
We will try to convert the problem in Eq.
(3) into a Bayesian version.
From the Bayesian viewpoint, we consider p w ( w ) as the prior of the coefﬁcients in the model.
In addition, we add the localization parameter r and its prior p r ( r ) to reﬂect the change introduced by each additional sample.
For any subset S ⊆ [ n ] , the distribution of all parameters and data X S , y S is:
[Equation]
The posterior distributions p ( w , r S | X S , y S ) and p ( y S , w , r S | X S ) differ by a regularization constant.
We ignore this regularization constant in the posterior distribution and only consider the main terms of the parameters.
We then formulate the Bayesian RLSR problem of searching for the subset and coefﬁcients by maximizing the log-posterior:
[Equation]
Note that we do not add any prior on σ 2 and only treat this as an adjustable parameter.
This is because, in the initial stage of the algorithm described in Section 3, the estimated σ 2 will be large due to the existence of outliers, and this will make the estimation of w excessively biased to the prior distribution.
This bias will be harmful, especially when the prior is not sufﬁciently accurate.
This phenomenon can be observed in Section 3.1.
To prove the positive effect of the prior on the RLSR problem, we require the properties of Subset Strong Convexity (SSC) and Subset Strong Smoothness (SSS) .
Given a set S ⊂ [ n ] , X S :
=
[ x i ∈ S ] ∈ R d ×| S | signiﬁes the matrix with columns in the set S .
The smallest and largest eigenvalues of a square symmetric matrix X are denoted by λ min ( X ) and λ max ( X ) .
Deﬁnition 1 (SSC Property) .
A matrix X ∈ R
d × n is said to satisfy the SSC property at level m with constant λ m if the following holds:
[Equation]
Deﬁnition 2 (SSS Property) .
A matrix X ∈ R
d × n is said to satisfy the SSS property at level m with constant Λ m if the following holds:
[Equation]
Input: Covariates X =
[ x 1 , ..., x n ] , responses y =
[ y 1 , ..., y n ] T , prior knowledge w 0 , penalty matrix M , corruption index k , tolerance ϵ
Output: solution ˆ w 1: b 0 ← 0 , t ← 0 ,
P MX ← X T ( XX T + M ) − 1 X , P MM ← X T ( XX T + M ) − 1 M 2: while ∥ b t − b t − 1 ∥ 2 > ϵ do 3: b t +1 ← HT k
( P MX b t
+ ( I − P MX )
y − P MM w 0 ) 4: t ← t + 1 ; 5: end while 6: return ˆ w ← ( XX T ) − 1 X
( y − b t )
These two properties are proposed in [2], and are intended to standardize the generation of the data matrix so that it will not be too abnormal.
They are used to prove the theorems in Section 4.
Methodology
We ﬁrst ignore the localization parameter r in Eq.
(5) and propose a simple method called TRIP in
Section 3.1.
TRIP demonstrates the effect of a prior on the hard thresholding method.
To improve the robustness and the accuracy of the estimation, the BRHT algorithm is proposed in Section 3.2.
3.1 TRIP: Hard Thresholding Approach to Robust Regression with Simple Prior
We propose a robust regression algorithm called TRIP (Algorithm 1), a hard Thresholding approach to Robust regression with sImple Prior.
In this subsection, only the prior p w ( w ) is considered and the localization parameter r is not added to the model.
We assume the variance σ 2 of ϵ i can be set by ourselves and that the prior p w ( w ) obeys a normal distribution N ( w 0 , Σ 0 ) , where w 0 and Σare determined in advance.
Through the above simple parameter settings, the problem in Eq.
(5) is transformed into the problem in Eq.
(1) with an additional regularization term:
[Equation]
where M = (Σ 0 /σ 2 ) − 1 .
To solve this problem, we are motivated by the hard thresholding method proposed by Bhatia [1], which concentrated on recovering the errors instead of selecting the ‘cleanest’ set.
The problem in Eq.
(8) can be formulated as min w ∈ R p , ∥
b ∥ 0 ≤ k ∗ 12 ∥ X T w − ( y − b ) ∥ 22 + 12 ( w − w 0 ) T M ( w − w 0 ) .
Thus, if we have an estimation ˆ b of the corruption vector b ∗ , the estimation of w ∗ can be easily obtained by ˆ w = ( XX T + M ) − 1 [ X ( y − ˆ b ) + M w 0 ] .
By substituting this estimation into the optimization problem, we obtain a new formulation of the problem:
[Equation]
where P MX = X T ( XX T + M ) − 1 X , P MM = X T ( XX T + M )
− 1 M .
The hard thresholding step in the TRIP algorithm can be viewed as b t +
=
HT k ( b t − ∇ f ( b t )) , where k is the selected corruption coefﬁcient.
The hard thresholding operator HT k is deﬁned as follows.
Deﬁnition 3 (Hard Thresholding) .
For any vector r ∈ R n , let δ − 1 r ( i ) represent the position of the i th element in r , which are arranged in descending order of magnitude.
Then, for any k < n , the hard thresholding operator is deﬁned as ˆ r =
HT k
( r ) , where ˆ r i =
r i if δ − 1 r ( i ) ≤ k and 0 otherwise.
The difference between the proposed TRIP algorithm and the original CRR [1] is the form of iteration step.
The iteration step in both TRIP and CRR can be expressed uniformly as HT k ( y − X T w t ) , but w t = ( XX T + M ) − 1 [ X ( y − b t ) + M w 0 ] in TRIP and w t =
( XX T ) − 1 X
( y − b t ) in CRR.
The w t in CRR is just a simple least square estimation, while the prior added in TRIP can be regarded to adding a quadratic regularization in each iteration.
This quadratic regularization can avoid the candidate of iteration that is too far from the prior mean, which is also helpful to ensure the
Input: Covariates X =
[ x 1 , ..., x n ] , responses y =
[ y 1 , ..., y n ] T , prior distribution p r ( r ) , p w ( w ) , corruption index k , tolerance ϵ
Output: solution ˆ w 1: b 0 ← 0 , t ← 0 , 2: while ∥ b t − b t − 1 ∥ 2 > ϵ do 3: w t ← V BEM ( X, y − b t , p r ( r ) , p w ( w )) 4: b t +1 ← HT k
( y − X T w t ) 5: t ← t + 1 ; 6: end while 7: return ˆ w ← ( XX T ) − 1 X ( y − b t ) numerical stability of solution.
Thus, as long as the prior is not mis-speciﬁed too much, TRIP will be more likely to identify the uncorrupted points, and the ﬁnal result of TRIP will be more robust than CRR.
Therefore, the prior plays an important role in TRIP. However, the weight of a prior in the solution depends entirely on the matrix M = (Σ 0 /σ 2 ) − 1 . Therefore, if we use an estimation of ˆ σ 2 to replace σ 2 , or give σ 2 a prior to calculate its posterior distribution, the overestimation of σ 2 will cause a severe increase in M in the initial iteration steps.
This will mislead the iteration and cause some deviation in the ﬁnal results.
To overcome this difﬁculty we can directly treat M as an adjustable parameter to control by specifying the form such as M = sI , where s is a positive number, and the suitable parameter can be chosen through 5-fold or 10-fold cross validation.
3.2 BRHT: Robust Bayesian Reweighting Regression via Hard Thresholding
In this subsection, we describe how the Bayesian reweighting method is combined with hard thresholding to give a more robust algorithm, BRHT (Algorithm 2), a robust Bayesian Reweighting regression via Hard Thresholding.
We ﬁrst introduce the reweighted probabilistic model (RPM) proposed in [24] for traditional linear regression.
For the covariates X and the response y , the RPM model can be formulated as follows:
[Equation]
where r is the local weight assigned to each sample, Z is the normalizing constant, ℓ ( y i | w , x i , σ 2 ) represents the likelihood of the normal distribution N ( x Ti w , σ 2 ) , and p w ( w ) , p r ( r ) are the priors of w and r , respectively.
By ignoring the normalizing constant, the problem in Eq.
(5) can be transformed into the following form under this RPM setting:
[Equation]
The speciﬁc form of the prior p r ( r i ) can be set to any nonnegative random variable distribution, including (but not limited to) the Gamma distribution, Beta distribution, or log-normal distribution.
Here, we still use the normal distribution N ( w 0 , Σ 0 ) as the form of p w ( w ) .
To solve the optimization problem in Eq.
(11), we use the two-step BRHT algorithm.
The key iteration step in BRHT is b t +1 ← HT k ( y − X T w t ) , where w t is calculated by maximizing the log-posterior of the RPM model:
[Equation]
However, the direct inference of Eq.
(12) is hard because of the nonconvexity of this problem.
In general, the parameters in this RPM model can be divided into two parts: the global variable w and the local latent variable r .
To solve the inference problem of RPM, a feasible method is to use variational Bayesian expectation maximization (VBEM).
We set q ( w , r ) =
q ( w ) q ( r ) to approximate the true posterior after several iterations of VBEM (Algorithm 3), and replace the estimation of w by the maximum a posteriori (MAP) estimation from the ﬁnal approximate posterior.
Full details of the VBEM method are given in Appendix A.
It is reasonable to ask why we are using Bayesian reweighting.
Note that the iteration step in the TRIP algorithm is b t
+1 ← HT k
( P MX b t
+ ( I − P MX )
y − P MM w 0 )
=
HT k ( y − X T w t ) , where w t =
( XX T + M ) − 1 [ X ( y − b t )
+ M w 0 ] .
Although we can show that TRIP already guarantees theoretical convergence, the estimation of w ∗ in every iteration still uses least-squares with a penalty term, which is easily affected by the corrupted points.
This disadvantage forces us to assign a higher weight to the prior to resist severe data corruption in the case of AAAs.
However, a higher weight on the prior means a larger estimation bias.
When applying the Bayesian reweighting method, the estimation in each step is more robust than the least-squares result, and thus the weight on the prior can be reduced to guide the iteration.
Therefore, the estimation bias is relatively small and the results are more robust.
This is reﬂected in the experimental results presented in Section 5.
We should also explain why we only use a prior for a few parameters.
It is important to ensure that the prior weights are neither too high nor too low.
As mentioned earlier, if we treat σ 2 as the parameter to be estimated, this places too much weight on the prior.
We also ensure that p w ( w ) does not create more uncertainty, such as setting p w ( w ) to p ( w , α ) =
N ( w 0 , α − 1 Σ 0 ) Gam ( α |
a α , b α ) .
Data corruption means that the subset of the training data may vary greatly from the prior information.
Thus, when calculating the posterior, the variance of w controlled by α will be very large to ﬁt the data, and so the prior information w 0 will have lower weights in the inference step.
The above problems also mislead the estimation and the selection of subsets, so we do not consider the uncertainty of these quantities and simply treat them as model parameters to be set in advance.
An adjustment method for all the parameters in BRHT is described in Appendix D.
Theoretical Convergence Analysis
In this section, we establish the convergence theory for the TRIP algorithm, and clearly explain how the prior effectively enhances the convergence of the RLSR model.
Theorems 1 and 2 summarize the results.
We also show the theoretical guarantee of the BRHT algorithm in Theorems 3–5, which further demonstrate the special properties achieved by using Bayesian reweighting.
Before presenting the convergence result, we ﬁrst introduce some notation.
Let λ t :=
( XX T + M ) − 1 X ( b t − b ∗ ) , g :=
( I − P MX ) ϵ , and f :=
P MM ( w ∗ − w 0 ) .
Let S t :=
[ n ] \ supp ( b t ) be the chosen subset that is considered to be uncorrupted, and I t :=
supp ( b t ) ∪ supp ( b ∗ ) .
Theorem 1.
Let X =
[ x 1 , . . .
, x n ] ∈ R d × n be the given data matrix and y =
X T w ∗ + b ∗ + ϵ be the corrupted output with sparse corruption of ∥ b ∗ ∥ 0 ≤ k · n .
For a speciﬁc positive semideﬁnite matrix M , X satisﬁes the SSC and SSS properties such that [Equation]
∗ λ min ( XX T + M ) < 1 . Then, if k > k ∗ , it is guaranteed with a probability of at least 1 − δ that, for any ε, δ > 0 , ∥ b T 0 − b ∗ ∥ 2 ≤ ε + O ( e 0 ) + O ( √ Λ k + [Equation] λ
[Equation]
( M ) λ min ( XX T + M ) )
∥ w ∗ − w 0 ∥ 2 after T
= O (log( ∥ b ∗ ∥ 2 ε )) iterations of TRIP, where e
= O ( σ √ ( k + k ∗ ) log n δ ( k + k ∗ ) ) under the normal design.
Theorem 2.
Under the conditions of Theorem 1, and assuming that x i ∈ R d are generated from the standard normal distribution, if k > k ∗ , it is guaranteed with a probability of at least 1 − δ that, for any ε, δ > 0 , the current estimation coefﬁcient w T 0 satisﬁes ∥ w T 0 − w ∗ ∥ 2 ≤ O ( 1 √ n )( ε + e 0 ) +
O ( [Equation]
[Equation] 2 after T
= O (log( ∥ b ∗ ∥ 2 ε )) steps.
For positive semi-deﬁnite matrices XX T and M , λ min ( XX T + M ) ≥ λ min ( XX T ) + λ min ( M ) .
Thus, the conditionΛ
[Equation]
XX T + M ) < 1 in Theorem 1 is weaker than the condition
[Equation]
1 of Lemma 5 of Bhatia [1], which shows that a prior can effectively improve the convergence of the algorithm.
Assigning a higher weight to a prior means that M has larger eigenvalues, so that the convergence condition will be more easily satisﬁed.
As a result, the TRIP algorithm can tolerate a higher proportion of outliers than the CRR method of Bhatia [1] and achieves a higher breakdown point.
In fact, under the condition lim n →∞ λ min ( M ) n = ξ , we can give an approximate expression of the breakdown point for TRIP when ξ is not too large: k ∗ ≤ k ≤ (0 .
3023 − √ 0 .
0887 − 0 .
0040 ξ ) n .
Details can be found in Appendix C.2.
However, the improved convergence comes at the cost of an unavoidable reduction in precision.
This can be seen from Theorem 2.
If the data corruption is such that k ∗ is O ( n ) and the maximum eigenvalue of M is also O ( n ) , then the bias of ˆ w cannot be decreased by adding more samples, which shows that there is a trade-off between convergence and accuracy.
A reliable prior improves both accuracy and convergence because it has a higher weight.
However, an inaccurate prior can also be helpful as long as it is quite different from the distribution of outliers, and the convergence can be improved through a prior with a low weight.
To prove the properties of our BRHT algorithm, we deﬁne the following two intermediate variables to simplify the description:
[Equation]
Theorem 5.
Suppose that the prior of r i is independently and identically distributed (iid).
We consider the t th iteration step of the BRHT algorithm, where w t , r t = arg max w ∈ R d , r ∈ R n + M ( w , r , b t ) and b t =
HT k
( y − X T w t − 1 ) is obtained from the hard thresholding step.
Then, we have that
U ( w t , r t , S t +1 ) ≥ U ( w t − 1 , r t − 1 , S t ) .
Theorem 6.
Consider a data matrix X and a speciﬁc positive semi-deﬁnite matrix M satisfying the SSC and SSS properties such thatΛ k
+ k ∗ λ min ( XX T + M ) < 1 .
Then, there exist α > 0 and 0 < γ ≤ 1 + ϵ , where ϵ is a small number, such that if k > k ∗ and Σ in the prior p w ( w ) is ασ 2 M − 1 , it is guaranteed with a probability of at least 1 − δ
that, for any ε, δ > 0 , ∥ b T 0 − b ∗ ∥ 2 ≤ ε +
O ( e 0 ) + O (
[Equation]
) γ ∥ w
[Equation]
− w 0 ∥ 2 after T {+
+}
= O (log( γ ∥ b ∗ ∥ 2 ε )) iterations of BRHT, where e
=
O ( σ √ ( k + k ∗ ) log n δ ( k + k ∗ ) ) under the normal design.
Theorem 7.
Under the conditions of Theorem 4 and with x i ∈ R d generated from the standard normal distribution, there exist α > 0 and 0 < γ ≤ 1 + ϵ , where ϵ is a small number, such that if k > k ∗ and Σ in the prior p w ( w ) is ασ 2 M − 1 , it can be guaranteed with a probability of at least 1 − δ that, for any ε, δ > 0 , the current estimation coefﬁcient w T 0 satisﬁes ∥ w T 0 − w ∗ ∥ 2 ≤
O ( 1 √ n )( ε + e 0 ) + O ( √ k + k ∗ λ max ( M ) n 3 /) γ ∥ w ∗ − w 0 ∥ 2 after T 0 = O (log( γ ∥ b ∗ ∥ 2 ε )) steps.
Theorem 5 shows that our BRHT algorithm is reasonable because it optimizes the problem in Eq.
(11) in each step.
Theorems 6 and 7 guarantee the convergence of the parameter, which means that if we introduce a prior p w ( w ) =
N ( w 0 , Σ 0 ) to the TRIP algorithm, it is convergent.
There then exists some α > 0 such that the prior p w ( w ) =
N ( w 0 , α Σ 0 ) in the BRHT algorithm guarantees convergent parameters and the bias of the estimation of w ∗ will be γ times that of TRIP.
Note that α is usually relatively large in practice, which causes a lower prior weight in BRHT.
Thus, when the convergence can be guaranteed, the bias of the estimator ˆ w can be signiﬁcantly reduced because
BRHT assigns a lower weight to the prior.
Even if the data are seriously corrupted, BRHT can ensure good results without signiﬁcant error.
All proofs of these theorems are given in Appendix C.
Experiments
In this section, we ﬁrst consider how to effectively ‘corrupt’ the dataset using two different attacks:
OAA and AAA.
We then report an extensive experimental evaluation to verify the robustness of the proposed methods.
Input: Covariates X =
[ x 1 , ..., x n ] , responses y =
[ y 1 , ..., y n ] T , true parameter w ∗ penalty coefﬁcient δ , corruption index k , tolerance ϵ
Output: solution ˆ w 1: b 0 ← 0 , t ← 0 ,
P δX ← X T ( XX T − δI ) − 1 X , P δ ← X T ( XX T − δI ) − 1 δI 2: while ∥ b t − b t − 1 ∥ 2 > ϵ do 3: b t +1 ← HT k
( P δX b t + ( I − P δX ) y + P δ w ∗ ) 4: t ← t + 1 ; 5: end while 6: ˆ w ← ( XX T ) − 1 X ( y − b t ) 7: C ← supp ( b t ) 8: return y C = X
TC ˆ w
5.1 Data and Metrics
In our experiments, the data generation can be divided into two steps.
First, we generate the basic model.
The true coefﬁcient w ∗ is chosen to be a random unit norm vector.
The covariant x i are iid in N (0 , I d ) .
The data are generated by y i =
x Ti w ∗
+ ϵ i , where ϵ i are iid in N (0 , σ 2 ) .
We set σ =in the experiments.
The second step is to generate the corrupted data using two kinds of attacks: OAA and AAA, as described in Section 5.2.
The aim is to produce k ∗ corrupted responses in the whole dataset.
The prior coefﬁcient w 0 is generated by w ∗ + ν u , where u is a random unit norm vector and ν is a non-negative number ( ν is set to 0.5 unless otherwise stated).
Σ 0 takes the form sI , where s takes a different value for each method.
All parameters are ﬁxed in each experiment.
Following the setting in [1], we measured the performance of the regression coefﬁcients by the standard L 2 error: r ˆ w = ∥
ˆ w − w ∗ ∥ 2 .
To judge whether the algorithm had converged, we used the termination criterion ∥ w t +1 − w t ∥ 2 ≤ 10 − 4 .
All results were averaged over 10 runs.
5.2 Corruption Methods
To demonstrate the efﬁciency of our proposed methods, we apply two different attacks to the dataset:
OAA and AAA.
The details of these two attacks are shown as follows
OAA : The set of corrupted points S is selected as a uniformly random k -sized subset of [ n ] , and the corresponding response variables are set as y i
=
x Ti w ∗ + b i
+ ϵ i , where b i are sampled from the uniform distribution U [0 , 10] and the white noise ϵ i ∼ N (0 , σ 2 ) .
AAA :
We use all information from the true data distribution to corrupt the data, and propose an adaptive data corruption algorithm (ADCA).
This algorithm is quite similar to TRIP; full details of ADCA (Algorithm 4) are given in Appendix B. δ is set to 0 .
1 n for n = 1000 , p = 200 , and to 0 .
2 n for n = 2000 , p = 100 .
Both TRIP and BRHT employ the prior p w ( w ) while the variance Σ 0 of p w ( w ) in OAA is set to be four times higher than that in AAA, which means that the prior has a higher weight in AAA.
Other parameters for these two algorithms are ﬁxed.
The prior distribution p r ( r ) is set to the Gamma distribution unless otherwise stated.
We also design another leverage point attack (LPA) to test the robustness of our methods.
More results can be seen in Appendix E.
5.3 Methods Comparison
Our methods are compared with three baselines: 1) CRR [1] is an effective robust regression method in cases where there are large numbers of random outliers; 2) Reweighted robust Bayesian regression (RRBR) [24] allows us to judge whether our proposed methods are better than the original method; 3) Rob-ULA [3] is an effective robust Bayesian inference method that approximately converges to the real posterior distribution in a ﬁnite number of steps in the presence of outliers.
The parameters of RRBR are the same as in the BRHT algorithm, except for the hard thresholding part.
For more comparisons of other [Figure] [Figure]
5.4 Recovery Properties of Coefﬁcients and Uncorrupted Sets
CRR, RRBR, and Rob-ULA are excellent robust regression methods or Bayesian inference methods, but they all show their limitations in the face of different kinds of attacks.
CRR achieves the best performance in the face of OAAs because it is theoretically unbiased, but it collapses rapidly when facing AAAs, as shown in Figures 1(c) and 1(d).
RRBR and Rob-ULA take priors into consideration, but RRBR cannot resist AAAs and Rob-ULA produces poor results under OAAs, as shown in Figures 1(a)–1(d).
TRIP produces a good effect against AAAs, while BRHT is not only optimal against AAAs, but also displays a similar effect to CRR in the case of OAAs.
This shows that BRHT is the most robust algorithm among those compared in this experiment.
The TRIP and BRHT algorithms are compared in Figures 2(a) and 2(b).
The TRIP method incorporates too much prior information in the case of OAAs, resulting in a greater estimation error than those of BRHT and CRR as shown in Figure 2(a).
However, under AAAs, the prior information of
TRIP and BRHT is enhanced by a factor of four, as described in Section 5.2.
BRHT converges under a weaker prior, while TRIP becomes trapped around a local optimum.
This shows that BRHT only needs to integrate weak prior information to ensure convergence.
Figures 2(c) and 2(d) illustrate the convergence properties under different weights of the prior p r ( r ) and coefﬁcient prior p w ( w ) .
Figure 2(c) shows that BRHT is not especially sensitive to the weight prior p r ( r ) when this prior is relatively reliable.
A log-normal distribution is the best choice when the prior p w ( w ) is relatively close to the real parameters and a Gamma distribution is more robust when the prior is imprecise, as shown in Figure 2(d).
Conclusion
This paper has described a novel robust regression algorithm named TRIP that achieves strong results in terms of resisting AAAs.
By adding a prior to the robust regression via hard thresholding, the recovery of coefﬁcients is signiﬁcantly improved.
Another algorithm, named BRHT, was designed to improve the robustness of TRIP and reduce the estimation error through the use of Bayesian reweighting regression.
We prove that both algorithms have strong theoretical guarantees and that the algorithms converge linearly under a mild condition.
Extensive experiments have illustrated that our algorithms outperform benchmark methods in terms of both robustness and efﬁciency.
There are several interesting future directions to extend current work.
Firstly, in this article, we only consider the case when y is corrupted.
One would consider using the prior information to better deal with the problem where both y and X are corrupted.
Secondly, it would be also interesting to further reduce the effect of a prior on the estimation to make it consistent.
Acknowledgments and Disclosure of Funding
This work was partly supported by National Key Research and Development Program of China (2021YFA1000300 and 2021YFA1000301), National Center for Mathematics and Interdisciplinary
Sciences, and Key Laboratory of Systems and Control of CAS.