Byzantine-Tolerant Distributed Multiclass Sparse Linear Discriminant Analysis
Abstract
Communication cost and security issues are both important in large-scale distributed machine learning.
In this paper, we investigate a multiclass sparse classiﬁcation problem under two distributed systems.
We propose two distributed multiclass sparse discriminant analysis algorithms based on meanaggregation and median-aggregation under the normal distributed system or Byzantine failure system.
Both of them are computation and communication efﬁcient.
Several theoretical results, including estimation error bounds, and support recovery, are established.
With moderate initial estimators, our iterative estimators achieve a (near-)optimal rate and exact support recovery after a constant number of rounds.
Experiments on both synthetic and real datasets are provided to demonstrate the effectiveness of our proposed methods.
INTRODUCTION
Multiclass classiﬁcation is one of the most important topics in machine learning and plays a crucial role in many ﬁelds, such as facial recognition, text mining, and gene analysis [Heisele et al., 2001, Zhang et al., 2015, Ramaswamy et al., 2001].
Linear discriminant analysis (LDA) is a useful tool in classiﬁcation problem, which aims to ﬁnd linear discriminant directions to separate the samples from different classes.
We consider the random variable X and its label Y , where
X ∈ R p is a multivariate normal random variable with mean µ k and covariance matrix Σ when Y = k for k = 1 , 2 , ..., K .
Let π k = P
( Y = k ) be the prior probability that variable X is observed from Class k , the oracle
Bayes rule under the LDA model can be written as
[Equation]
arg
[Equation]
(cid:26)
[Equation]
, where θ ∗ k = Σ − 1 ( µ k − µ 1 ) for k = 1 , 2 , ..., K denotes
Fisher’s discriminant directions.
In practice, we need to estimate µ 1 , µ k , and Σ to obtain the estimation of θ ∗ k .
Given independent samples { ( X i , Y i ) , i = 1 , 2 , ..., N } from K classes and denote (cid:80) Ni =
I
( Y i = k ) =
N k , the classical estimators of Fisher’s discriminant directions are given by (cid:98)
[Equation]
1 ) for k = 2 , ..., K where
[Equation]
Then the classical discriminant rule is
[Equation]
where (cid:98)
π k = N k /N .
It has been shown to be both theoretical and practical efﬁcient in the classical ﬁxed dimensionality regime.
Nevertheless, the classical linear discriminant rule performs poorly (no better than random guessing) when the dimensionality p increases as the sample size N [Bickel and
Levina, 2004].
The main reason is that the sample covariance matrix will be ill-conditioned in such a case.
Another related problem is over-ﬁtting, and it leads to great performance loss to the model.
Due to the rapid growth in the size of datasets and resource sharing, there has been tremendous interest in developing distributed machine learning methods in recent years.
However, not many works focus on sparse LDA in a distributed environment.
The main challenge of distributed estimation is the communication cost.
The existing sparse LDA algorithms require constructing an overall sample covariance matrix, which is unrealistic in the distributed system when p is large since the bandwidth of the local machine is limited.
To reduce communication cost, Tian and Gu [2017] proposed a distributed sparse LDA algorithm that only required one round of communication.
To the best of our knowledge, it is also the only existing distributed sparse LDA algorithm.
Despite this, the computational issue is salient for each local machine.
The algorithm involves solving largescale linear programming and estimating the inverse of the covariance matrix, which is computation expensive in the high-dimensional setting.
Furthermore, the convergence rate in Tian and Gu [2017] will be sub-optimal when the number of local machines is large.
Another concerning issue in distributed machine learning is security.
Most distributed machine learning algorithms require a master machine to aggregate the information from local machines, which are susceptible to errors due to unpredictable and potential attacks.
The security issue is more prominent in large-scale distributed systems, such as Federated Learning [Koneˇcn ` y et al., 2016].
Byzantine failure is used to model the local machine’s inherent abnormal behavior, which means that some local machines may send wrong messages or behave completely arbitrarily [Lamport et al., 1982].
The algorithm in Tian and Gu [2017] takes simple averaging aggregation of the information from local machines, which is highly non-robust in the Byzantine failure system.
We note that there are several works related to the robust LDA algorithm
[Zhang and Yeung, 2010, Wen et al., 2018, Nie et al., 2019], whereas these methods only work for heavy-tailed data and are not resistant to Byzantine failure.
Thus it is of great interest to develop Byzantine-robust multi-classiﬁcation algorithm with a theoretical guarantee.
Although there are extensive studies for distributed estimation and optimization, few are related to sparse LDA in the high-dimensional regime.
Tian and Gu [2017] proposed a communication efﬁcient distributed sparse LDA method by constructing a debiased version of linear programming discriminant (LPD) estimator [Cai and Liu, 2011] for the binary classiﬁcation task.
Recently, Bian et al.
[2020] proposed a distributed sparse LDA method that does not require the communications of data information among different local machines.
Unfortunately, no theoretical guarantee was provided in their work.
More importantly, both of them are sensitive to the abnormal behaviors of local machines in the distributed system.
1.1 OUR CONTRIBUTIONS
To address the challenge of increasing dimensionality and
Byzantine failure, we propose a new communication efﬁcient distributed sparse LDA algorithm for distributed multiclassiﬁcation problem in two different systems, respectively:
• System
I: The distributed system without Byzantine failures; • System II: There are α fraction Byzantine local machines, and the remaining 1 − α fraction local machines are normal.
Under System I, we propose the distributed sparse LDA method based on mean-aggregation ( Mean-DSLDA ).
As for System II, the median-aggregation is applied against the potential Byzantine failure.
Thus we propose the Byzantinetolerant Median-DSLDA .
With these two methods, we highlight the main contributions of this paper:
Our proposed algorithm shares the same O ( p ) communication cost with the state-of-the-art distributed learning algorithms [Lee et al., 2017a, Wang et al., 2017].
Compared with Tian and Gu [2017], our proposed algorithm requires less computation complexity in the local machine.
The theoretical results guarantee that our proposed algorithm attains (near-)optimal statistical convergence rate and exact support recovery after a constant number of communication rounds.
The experiments on synthetic and real data show that our proposed algorithm converges quickly, and Median-DSLDA is highly robust to Byzantine failures.
1.2 RELATED WORK
Sparse LDA methods in the high-dimensional regime have been broadly investigated in recent years [Witten and Tibshirani, 2011, Cai and Liu, 2011, Fan et al., 2012, Mai et al., 2012, 2019].
For binary classiﬁcation case, Cai and Liu [2011] and Mai et al.
[2012] assumed the Bayesian discriminant direction Σ − 1 ( µ 2 − µ 1 ) is sparse then directly estimated it by using Dantzig selector [Candes and Tao, 2007] and lasso penalty respectively.
Unfortunately, these methods can not be generalized to multiclass classiﬁcation cases easily.
For the multiclass sparse LDA problem, penalized Fisher’s discriminant [Witten and Tibshirani, 2011], sparse optimal scoring [Clemmensen et al., 2011] and multiclass sparse discriminant analysis (MSDA) proposed in Mai et al.
[2019] are three popular proposals.
Speciﬁcally, the MSDA method simultaneously estimates all the sparse Bayesian discriminant directions by solving a quadratic group lasso problem.
Owing to the growth of sample size and dimensionality of datasets, extensive works on high-dimensional sparse distributed machine learning algorithms are proposed.
A popular method for distributed sparse estimation is divideand-conquer debiased (DC-debiased) framework proposed by Lee et al.
[2017a].
Thanks to easy implementation and low communication cost, the DC-debiased scheme has been broadly deployed in several sparse estimation problems [Lv and Lian, 2022, Tian and Gu, 2017, Battey et al., 2018, Lian and Fan, 2018].
However, the debiased operation requires estimating the inverse of the Hessian matrix, which leads to expensive computation costs for each local machine.
There exists a constraint on the number of local machines for DC-debiased estimators to achieve optimal statistical convergence rate[Zhang et al., 2013, Lee et al., 2017b, Battey et al., 2018].
Wang et al.
[2017] and Jordan et al.
[2019] developed another framework, namely Communication-efﬁcient Surrogate Likelihood (CSL), which reﬁnes the estimator by multi-round communication.
Each local machine in the CSL framework only needs to compute and transmit gradients in each round, and then the master machine solves a penalized sub-problem.
Particularly, this kind of method has no restriction on the number of machines.
Relate literature based on CSL scheme are Wang et al.
[2019], Fan et al.
Chen et al.
To tackle potential Byzantine failures in distributed learning, some related works are proposed by letting the master machine conduct a robust aggregation on the gradient information received from local machines [Yin et al., 2018, Xie et al., 2018, Alistarh et al., 2018, Li et al., 2019, Yin et al., 2019].
The most common robust aggregation rule is medianof-means (MOM).
Yin et al.
[2018] established several optimal statistical rates under mild conditions for the proposed robust algorithms based on median and trimmed mean operations.
Other robust aggregation rules such as marginal trimmed mean, dimensional median, Krum are also investigated in the existing literature [Xie et al., 2018, Li et al., 2019].
Tu et al.
[2021b] proposed a variance-reduced version of the median-of-means aggregation procedure motivated by the composite quantile.
For distributed penalized regression problems, Tu et al.
[2021a] developed a Byzantine-robust least-square Lasso method.
1.3 NOTATION
The following notations will be used throughout the paper.
For a vector x ∈ R p , (cid:107) x (cid:107)
=
(cid:80) pj
=1 | x j | , (cid:107) x (cid:107)
=
( (cid:80) pj =1 x 2 j ) 1 / 2 and (cid:107) x (cid:107) ∞ =
max j | x j | .
For a matrix
A =
( A i,j
) ∈ R p × p , spectral norm is deﬁned by (cid:107) A (cid:107) 2 = sup x ∈ R p (cid:107) Ax (cid:107) 2 , (cid:96) ∞ norm is deﬁned by (cid:107) A (cid:107) ∞
=
max i (cid:80) pj =1 | A i,j | and | A | ∞ = max i,j | A i,j | .
For symmetric matrix A , the smallest and largest eigenvalue of A are denoted by λ min ( A ) and λ max ( A ) respectively.
For a matrix A ∈ R m × n , A ST denotes the submatrix ( a s i t j ) for
S =
{ s 1 , . . .
, s r } ⊆ { 1 , . .
, m } and T = { t 1 , . . .
, t q } ⊆ { 1 , . . . , n } .
For two sequences of positive numbers c n and d n , we write c n (cid:46) d n if there exists some positive constant c such that c n ≤ cd n holds for sufﬁciently large n; and c n (cid:16) d n if c n (cid:46) d n and d n (cid:46) c n .
For a sequence of random variables X n , X n = O P ( d n ) means that for any ε > 0 there exists some positive constant C such that P ( | X n | > Cd n ) < ε .
MODELS AND ALGORITHMS
Recall that the Bayesian discriminant directions to be estimated are θ ∗ k = Σ − 1 ( µ k − µ 1 ) for k = 2 , ..., K .
According to Mai et al.
[2019], the contribution to discriminant from j -th variable of X vanishes if and only if θ ∗ 2 ,j = · · · = θ ∗ K,j = 0 , which means θ ∗ k,j , k = 2 , ..., K are grouped according to j .
Denote the support set S to be
S =
{ j : θ ∗ kj (cid:54) = 0 for some k = 2 , ..., K − 1 } and s = | S | is the sparsity.
Given independent samples { ( X i , Y i ) , i = 1 , 2 , ..., N } from K classes, Mai et al.
[2019] proposed the multiclass sparse discriminant analysis (MSDA) method and estimated θ ∗ k for k = 2 , ..., K simultaneously by solving the following group lasso problem
[Equation]
where θ ( j ) =
( θ 2 ,j , . .
, θ K,j ) T and λ > 0 is the tuning parameter.
MSDA achieves variable selection consistency in the centralized sample case.
For new observation X new , we classify X new to Class (cid:98) Y if
[Equation]
where (cid:98) θ 1 = 0 .
For the ease of presentation, we suppose that all the samples { ( X i , Y i ) , i = 1 , 2 , ..., N } are stored in the master machine and M local machines evenly.
Denote the sample index in the m -th machine by H m for m = 0 , 1 , ..., M where H 0 is the master machine, then the samples in the m -th machine are { ( X i , Y i ) : i ∈ H m } .
When Y i =
k , the corresponding observation X i is sampled from multivariate normal distribution N ( µ k , Σ ) for k = 1 , 2 , ..., K .
Without loss of generality, we assume the samples from
K classes are evenly distributed in both the master machine and M local machines.
Thus the sample size of Class k in each machine is n k .
Let n = (cid:80) K k =1 n k be the sample size of the master machine then N = n ( M + 1) and
N k = n k ( M + 1) .
In System II, owing to the existence of Byzantine failure machines, we assume that the master machine can never be corrupted so that we can trust the information collected in it.
For the remaining local machines, some of them may collect contaminated data or send arbitrary wrong values to the master machine.
Among the
M local machines, the fraction of Byzantine machines is denoted by α and the remaining 1 − α local machines are normal.
In each machine H m for m = 0 , 1 , ..., M , we compute the corresponding estimators (cid:98) π m,k = (cid:80) i ∈H m I ( Y i = k ) /n , (cid:98) µ m,k = (cid:80) { i ∈H m : Y i =
k } X
i /n
k and (cid:98) Σ m = 1 n
K (cid:88) k
= (cid:88) { i ∈H m : Y i = k } (cid:0) X
[Equation]
(cid:98)
[Equation]
m
[Equation]
k (cid:1) (cid:0) X
[Equation]
(cid:98) µ m,k (cid:1) (cid:62) .
Given initial estimators (cid:98) θ (0) k for k = 2 , ..., K and motivated by CSL framework [Jordan et al., 2019], we update the estimator in the t -th iteration by solving the following quadratic group lasso problem
[Equation]
where b ( t − 1) k is a consistent estimator of Σ (cid:98) θ ( t − 1) k
[Equation]
µ 1 ) given (cid:98) θ ( t − 1) k .
The optimization problem (2.1) can be efﬁciently solved by several well studied methods, such as group coordinate descent algorithm
[Yuan and Lin, 2006] and iterative soft-thresholding algorithm
[Beck and
Teboulle, 2009].
Thus the keystone is to construct b ( t − 1) k in the master machine by using the local information under different distributed systems (System I and II) efﬁciently and safely.
The following are two kinds of aggregations considered in our algorithms to construct b ( t − 1) k and estimate µ k , π k .
Aggregation 1 (Mean-aggregation) .
The master machine estimate µ d and π d by using (cid:98) µ d = (cid:80)
Mm =0 (cid:98) µ m,d / ( M + 1) and (cid:98) π
d = (cid:80) Mm =0 (cid:98) π m,d / ( M + 1) respectively for d = 1 , 2 , ..., K .
Denote (cid:98) d ( t − 1) m,k =
[Equation]
n
[Equation]
d
[Equation]
i :
[Equation]
X
[Equation]
(cid:98)
[Equation]
) k , and construct b ( t − 1) k by using (cid:98) b ( t
[Equation] [Equation]
d ( t
[Equation]
− ( (cid:98)
[Equation]
(cid:98) µ 1 ) , where (cid:98) d ( t − 1) k
= (cid:80)
Mm =0 (cid:98) d ( t − 1) m,k / ( M + 1) .
For vectors x m ∈ R p , m = 0 , 1 , ..., M , the coordinatewise median operator is denoted by cmed .
Then z := cmed { x m , m = 1 , 2 ..., M } is a p -vector and z j is the median of { x m,j : m = 0 , 1 , ..., M } .
Then we deﬁne the following aggregator and estimators.
Aggregation 2 (Median-aggregation) .
The master machine estimate µ d and π d by using (cid:101) µ d = cmed { (cid:98) µ m,d : m = 0 , 1 , ..., M } and (cid:101)
π d = median { (cid:98) π m,d : m = 0 , 1 , ..., M } respectively for d = 1 , 2 , ..., K .
Denote (cid:101) d
( t − 1) m,k = (cid:98) Σ m (cid:98) θ
( t − 1) k and construct b ( t − 1) k by using (cid:101) b ( t − 1) k
= (cid:101) d
( t − 1) k − ( (cid:101) µ k − (cid:101) µ 1 ) , where (cid:101) d ( t − 1) k
= cmed { (cid:101) d ( t − 1) m,k : m = 0 , 1 , ..., M } .
With the help of Aggregation 1 and 2, we propose
Mean-DSLDA and Median-DSLDA under System
I and
System II, respectively.
We start by obtaining (cid:98) µ k , (cid:98) π k , (cid:101) µ k , and (cid:101) π k in the master machine through one round of communication.
Given the initial estimators (cid:98) θ (1) k satisfying some mild conditions (see Section 3), the vectors (cid:98) d (1) m,k or (cid:101) d (1) m,k are parallelly computed on the m -th local machine.
We only need to communicate these p -dimension vectors to the master machine, thus the communication cost is O ( p ) .
With these constructed vectors, the master machine computes b (1) k by Aggregation 1 and 2 under System I and II respectively and obtains the updated estimators (cid:98) θ (1) k by solving (2.1).
These steps can be repeated iteratively to reﬁne the estimators at each communication round.
Algorithm 1: Distributed Multiclass Sparse Linear Discriminant Analysis (DSLDA)
Input: Local data sets { X i , Y i : i ∈ H m } for m = 0 , 1 , ..., M , the number of iterations T , the initial estimators { (cid:98) θ (0) k : k
= 2 , ..., K } , the tuning parameters λ t for t = 1 , ..., T .
Output: Final estimators { (cid:98) θ
( T ) k : k
= 2 , ..., K } .
for m = 0 , 1 , ..., M do
The m -th machine : Compute (cid:98) µ m,k , (cid:98) π m,k then send them to the master machine.
endThe master machine: Compute (cid:98) µ k , (cid:98) π k , (cid:101) µ k and (cid:101)
π k for k = 1 , 2 , ..., K then broadcast (cid:98) θ (0) k and (cid:98)
µ k to all local machines.
for t = 1 , 2 , ..., T do for m = 0 , 1 , ..., M do
The m -th machine : Compute
[Equation]
System II , according to Aggregation 1 and 2 then send it to the master machine.
end
The master machine : Construct b ( t − 1) k by b ( t
[Equation]
, System I (cid:101)
[Equation]
[Equation]
[Equation]
System II , according to Aggregation 1 and 2 and obtain (cid:98) θ ( t ) k by solving (2.1).
Then broadcast (cid:98) θ ( t ) k for k = 2 , ..., K to all local machines.
end
After T communication rounds, we can obtain the ﬁnal estimators (cid:98) θ ( T ) k for k = 2 , ..., K .
Then for new observation
X new , we classify X new to Class
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
The details of Mean-DSLDA and Median-DSLDA are described in Algorithm 1.
Our proposed algorithm and the
DC-debiased algorithm in Tian and Gu [2017] both require computing and storing the local covariance matrix (cid:98) Σ m in each local machine.
In addition to this operation, the DCdebiased algorithm also needs to estimate the sparse discriminant direction and the inverse of the covariance matrix in each local machine, leading to O ( np 2 ) extra computation complexity least.
In each communication round of Algorithm 1, each local machine only needs to compute (cid:98) d ( t − 1) m,k or (cid:101) d ( t − 1) m,k .
The local computation complexity of our method is O ( Tp 2 ) , which is sufﬁciently reduced compared with the
DC-debiased algorithm since T is a constant based on our theory (see the discussion of Corollary 3.1).
THEORETICAL RESULTS
In this section, we present the theoretical results of our proposed Mean-DSLDA and Median-DSLDA including the estimation error bounds and support recovery.
With slightly abusing notations, we denote
[Equation]
, and
[Equation]
Let Z ∗ j ∈ R K − 1 be the subgradient of (cid:107) θ (cid:107) 2 evaluated at θ ∗ ( j )
= ( θ ∗ 2 ,j , . .
. , θ ∗ K,j ) T and Z ∗ S,k
=
( Z ∗ j,k : j ∈ S ) .
Before presenting the formal results of our proposed method, we introduce the following technical assumptions for the clarity of the theoretical guarantee.
( C 1)
There exists a positive constant c ≥ 1 such that c − 1 ≤ λ min ( Σ ) ≤ λ max ( Σ ) ≤ c .
There exist some constants c 1 > 0 and c 2 < ∞ such that ∆ min > c 1 , ∆ max < c 2 .
( C 2)
The sample size of each class satisﬁes N 1 (cid:16) N 2 (cid:16) · · · (cid:16) N K .
The dimensionality p satisﬁes log p =
O ( n ν ) with ν < 13 .
The sparsity s satisﬁes that s = O ( n β ) with β < 13 .
The sample size of the master machine satisﬁes that n (cid:38) N ψ with 0 < ψ < 1 .
( C 3)
The initial estimators (cid:98) θ (0) k for k = 2 , ..., K have the common support set (cid:98) S (0) and satisfy that max 2 ≤ k ≤ K (cid:107) (cid:98) θ (0) k − θ ∗ k (cid:107)
= O P ( a n ) with a n = o (1) .
Moreover, assume that P ( (cid:98) S (0) ⊆ S )
→ 1 .
( C 4) Suppose that Σ satisﬁes that (cid:107) Σ S c S Σ − 1 SS (cid:107) ∞ < ∞ and for some κ ∈ (0 , 1) ,
[Equation]
( C 5)
The
fraction of Byzantine local machines α <
Condition ( C 1 ) is common in sparse LDA literatures [Shao et al., 2011, Cai and Liu, 2011, Mai et al., 2012].
Condition ( C 2 ) is considered when establishing the support recovery consistency results, which also appears in Mai et al.
From condition ( C 3 ), the dimension p is allowed to be greater than the local sample size n .
Condition ( C 4 ) can be easily satisﬁed if we choose some sparse estimators obtained by local samples as the initial estimators.
Condition ( C 5 ) guarantees the statistical consistency of the medianaggregation against Byzantine failures, similar assumption can be found in Yin et al.
[2018], Tu et al.
[2021a,b].
3.1 ESTIMATION ERROR BOUND
Theorem 3.1.
Suppose that conditions ( C 1 )-( C 3 ) and ( C 5 ) hold.
By choosing the tuning parameter λ t =  
[Equation]
System I
[Equation]
for some sufﬁciently large positive constant C , we are guaranteed that
[Equation]
for k = 2 , ..., K under both System I and II.
Theorem 3.1 provides (cid:96) 2 estimation error bounds after the t -th iteration in Algorithm 1.
The ﬁrst term in (3.1) is the minimax rate of (cid:96) 2 error bound for (group)lasso estimators in the centralized sample case (see Raskutti et al.
Bühlmann and Van De Geer [2011], Wainwright [2019]).
The second term implies that the (cid:96) 2 estimation error converges geometrically to the optimal order with contraction rate (cid:112) s log p/n .
Note that Median-DSLDA has two additional terms in the convergence rate.
The term α (cid:112) s/n is owing to the existence of Byzantine failure machines while √ s/n results from the median-aggregation.
Therefore,
Theorem 3.1 also indicates that Mean-DSLDA is more efﬁcient than Median-DSLDA under System
I. Considering that mean-aggregation is not resistant to Byzantine failures,
Median-DSLDA is preferred under System II.
Corollary 3.1.
Under the same conditions and settings in
Theorem 3.1, if the initial estimator satisﬁes that
[Equation]
and the number of iteration round T satisﬁes
[Equation]
we are guaranteed that
[Equation]
System I
[Equation]
System II
In System I, there is no constraint on the number of local machines M since we only require the sample size of the master machine satisﬁes n ≥ N ψ .
For System II, we require all the local machines have O ( n ) samples to ensure the consistency of Median-aggregation.
In fact, if the number of local machines satisﬁes M (cid:46) √ N in the System II, the (cid:96) 2 error bounds of Median-DSLDA becomes
O P ( (cid:112) s log p/N + α (cid:112) s/n ) , which cannot be improved due to the existence of Byzantine machines.
In accordance with the assumption on the sparsity s , the right hand side of (3.2) can be bounded by log( N/n ) [Equation]
It connotes that our proposed method can achieve optimal convergence rate after a constant number of communication rounds.
It is worth comparing our results with Tian and Gu [2017] for the binary classiﬁcation case ( K = 2 ) under System I.
To achieve an optimal convergence rate, they required that the number of local machines M satisﬁes
M (cid:46) (cid:112) N/ log p/ max( s, s (cid:48) ) , where s (cid:48) is the maximum number of nonzero elements in each column of Σ − 1 .
However, under System I, the Mean-DSLDA algorithm has no constraint on the number of local machines and does not require the sparsity of Σ − 1 .
3.2 SUPPORT RECOVERY
Due to the group lasso property, (cid:98) θ ( t ) k for k = 2 , ..., K have the same support set.
We denote the common support set of estimator (cid:98) θ ( t ) k by (cid:98) S ( t ) for t = 1 , 2 , ..., T .
Theorem 3.2.
Suppose that conditions ( C 1 ) − ( C 5 ) hold, with the same choices of the tuning parameter λ t in Theorem 3.1, we have (cid:98) S ( t ) ⊆
S holds with probability tending to 1.
Moreover, suppose that there exists a sufﬁciently large constant C > 0 such that
[Equation]
then we have (cid:98) S ( t )
=
S with probability tending to 1 for both Mean-DSLDA and Median-DSLDA .
Theorem 3.2 guarantees the exact support recovery consistency of Algorithm 1 under the beta-min condition (3.3).
Note that the beta-min condition becomes weaker as iteration round t increases.
And if T satisﬁes (3.2), the beta-min condition of Mean-DSLDA in Algorithm 1 under System
I will reduce to θ ∗ min ≥ C (cid:107) Σ − 1 SS (cid:107) ∞ (cid:112) log p/N , which coincides with the order in Wainwright [2009], Mai et al.
SIMULATION RESULTS
In this section, we will investigate the numerical performance of the proposed Byzantine-tolerant distributed sparse
LDA method on synthetic data.
Three metrics are used to evaluate the performance of algorithms: the average (cid:96) 2 estimation error (cid:80) Kk =2 (cid:107) (cid:98) θ k − θ ∗ k (cid:107) 2 /K , the misclassiﬁcation rate and the F 1 score.
The F 1 score is deﬁned as
[Equation]
, where precision = | (cid:98) S ∩ S | / | (cid:98) S | and recall = | (cid:98) S ∩ S | / | S | and (cid:98) S is the support set of (cid:98) θ k .
In the following experiments, we generate a training set of size N , a validation set of size 1 , 000 and a test set of size 1 , 000 independently, then randomly partition the training set into M + 1 machines (including the master machine) evenly.
The validation set is used to choose tuning parameter and the test set is used to compute misclassiﬁcation rate.
All the results are averaged over 200 independent trails.
4.1 MULTI-CLASS TASK
The generation of synthetic data is as follows.
Denote the label of each class by k for k = 1 , 2 , ..., K .
We set K = 5 , p = 600 , β jk = 1 . 6 for j = 2 k − 1 , 2 k ; k = 1 , ..., K and β jk = 0 otherwise.
The covariance matrix is Σ = ( σ ij ) p × p where σ ij = 0 .
5 | i − j | .
Let µ k = Σ β k and θ ∗ k = β k − β 1 , then the support set S = { 1 , 2 , ..., 10 } .
In each machine, we generate Class k samples independently from N ( µ k , Σ ) for k = 1 , 2 , ..., K with equal sample size.
For Byzantine local machines, the label Y is replaced by 6 − Y .
The master machine is normal in our setting.
For comparison, we also run MSDA algorithm
[Mai et al., 2019]
using the centralized data, which is abbreviated as C-MSDA .
The results are recorded by taking averages over 100 independent trials.
In the ﬁrst experiment, we investigate the communication rounds of our proposed method needed to achieve numerical convergence.
We set the total sample size N as 50,000, and the number of machines (including the master machine) is 100.
In System II, the fraction of Byzantine local machines is α = 0 .
The trajectories of three evaluation metrics are presented in Figure 1, where the horizontal lines represent the results of C-MSDA .
As we can see, all metrics of Mean-DSLDA diverge under System II, and [Figure] 20 Iteration 0. L2 E
rr o r
[Figure]
[Figure]
Iteration [Figure] [Figure] [Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
f
[Figure]
a t
[Figure]
e
[Figure] [Table] Misclassiﬁcation rate F1 Score
System I
System
[Table]
System
[Table]
System
[Table]
System I
System
[Table] m [Table] > 0.
> 0.
> 0.
> 0. > 0. Median-DSLDA shows robustness against Byzantine failure. In addition, the evaluation metrics of Mean-DSLDA (under System I) and Median-DSLDA converge numerically within 5 communication rounds, which corroborates the statement in Corollary 3.1.
The difference of (cid:96) 2 error between our proposed two methods and C-MSDA is tiny, which indicates the performance loss caused by distributed estimation is negligible in two systems. In the second experiment, we investigate the effect of the number of local machines on our proposed algorithm. The local sample size is ﬁxed as n = 200 , and the number of machines (including the master machine) varies from 100 to 500. We summarize the averaged results in Table 1.
It implies that both Mean-DSLDA (under System I) and
Median-DSLDA are not sensitive to the number of machines since our proposed method can attain an optimal convergence rate without the constraint on the number of local machines.
In the third experiment, we run Median-DSLDA under
System II with the fraction of Byzantine local machines varying from 0 to 0.2.
The total sample size N is ﬁxed as 20,000, and the number of machines M +1 is 100.
We report averages and standard deviations of three evaluation metrics after the 5-th iteration in Table 2.
There is no signiﬁcant performance deterioration for Median-DSLDA with the increasing fraction of Byzantine local machines.
4.2 BINARY-CLASS TASK
For binary-class task, we compare our method with the debiased procedure in Tian and Gu [2017], which is abbreviated as DC-LPD .
For the fairness of comparison, we follow the same data generation regime in DC-LPD .
From the results in Table 3, it can be seen that our method has a better performance over DC-LPD , and the computational superiority is salient.
REAL DATA
In this section, we use the MNIST dataset 1 and ISOLET dataset 2 to verify the performance of our proposed algorithm in real data.
A brief description of the two datasets is given in Table 4.
We randomly divide the training sets
[Table] [Table] of the MNIST dataset and ISOLET dataset into 20 andmachines, respectively (including the master machine) with an equal sample size.
For Byzantine local machines, we use a similar adversarial setting in synthetic data experiments.
Then we conduct Algorithm 1 by setting the iteration step T = 20 .
The tuning parameter λ t in each iteration is selected by ﬁve-fold cross-validation.
[Table] The experiment results are reported in Figure 2.
As we can see, the test errors of our proposed methods decrease dramatically after the ﬁrst communication round, then becomes stable in the future iterations.
Under System I, the test error of Mean-DSLDA is lower than Median-DSLDA .
Under
System II, the classiﬁcation performance of Mean-DSLDA is severely affected by the Byzantine machines. In addition, the utility of Median-DSLDA does not degrade signiﬁcantly under System II.
CONCLUSIONS
In this paper, we proposed a communication efﬁcient distributed sparse linear discriminant analysis ( Mean-DSLDA ) algorithm
under a normal distributed system and
its
Byzantine-tolerant version ( Median-DSLDA ) for the multi-classiﬁcation problem.
Compared with the existing distributed sparse LDA algorithm, our proposed algorithm sufﬁciently reduces the computation complexity of each local machine.
To achieve the optimal statistical convergence rate, Mean-DSLDA does not require any restrictions on the number of local machines M , which can be applied in a large scale distributed system.
Experiments on synthetic and real data corroborate the theoretical results and the superiority of Median-DSLDA against Byzantine failures.
[Figure]
Acknowledgements
Weidong Liu’s research is supported by National Program on Key Basic Research Project (2018AAA0100704), NSFC
Grant No. 11825104 and 11690013, Youth Talent Support
Program, Shanghai Municipal Science and Technology Major Project (2021SHZDZX0102).
Xiaojun Mao’s research is supported by NSFC Grant No. 12001109 and 92046021, the Science and Technology Commission of Shanghai Municipality grant 20dz